Building your first aplication using Apache Apex

Jun 4, 20161 like479 views

This document provides an overview of building an Apache Apex application, including key concepts like DAGs, operators, and ports. It also includes an example "word count" application and demonstrates how to define the application and operators, and build Apache Apex from source code. The document outlines the sample application workflow and includes information on resources for learning more about Apache Apex.

Yogi Devendra
yogidevendra@apache.org
Building your first Apache Apex
Application

● Key concepts: DAG, Operators, Ports
● APIs for defining Applications, Operators
● “Word Count” example DAG
● Building Apache Apex from source code
● Creating a sample application
● Demo
● Questions
Outline

● An Application is defined as Directed Acyclic Graph : DAG
● Vertices of the DAG are computational units : Operators
● Edges of the DAG are data tuples in-motion : Streams
● Operator end-points for input , output : Ports
● An Operator takes one or more input streams, performs computations & emits one or more output streams
○ Each operator is USER’s business logic, or built-in operator from our open source library
○ Operator may have multiple instances that run in parallel
Application as a DAG

● MyApplication implements StreamingApplication
○ Provide implementation for populateDAG
○ Stitch the DAG
● SampleOperator extends BaseOperator
○ Define input ports, output ports
○ Define process methods
○ Optional : Define beginWindow, endWindow, setup,
teardown
APIs : Application, Operator

● Data at Rest - Count occurrences of words in a file
● Data in Motion - Emit counts at the end of the window
● Another variation - Emit cumulative counts at the end of
every window.
Sample application
Apex Application DAGHDFS
LOGS
Lines Counts

Defining DAG
Reader Parser Counter Output
Input
Operator
(Adapter)
Output
Operator
(Adapter)
Generic
Operators
HDFS
LOGS

• Java : 1.7.x
• mvn : 3.0 +
• git : 1.7 +
• Apache hadoop : How to : Single node cluster
• Apache Apex Core
• git clone git@github.com:apache/apex-core.git
• cd apex-core/
• git checkout master
• mvn clean install -DskipTests
• Apache Apex Malhar
• git clone git@github.com:apache/apex-malhar.git
• cd apex-malhar/
• git checkout master
• mvn clean install -DskipTests
• DataTorrent RTS community edition
Building Apache Apex

Apache Airflow is an open-source workflow management platform developed by Airbnb and now an Apache Software Foundation project. It allows users to define and manage data pipelines as directed acyclic graphs (DAGs) of tasks. The tasks can be operators to perform actions, move data between systems, and use sensors to monitor external systems. Airflow provides a rich web UI, CLI and integrations with databases, Hadoop, AWS and others. It is scalable, supports dynamic task generation and templates, alerting, retries, and distributed execution across clusters.

Airflow 101SaarBergerbest

The document provides an overview of Apache Airflow, an open-source workflow management platform for data pipelines. It describes how Airflow allows users to programmatically author, schedule and monitor workflows or data pipelines via a GUI. It also outlines key Airflow concepts like DAGs (directed acyclic graphs), tasks, operators, sensors, XComs (cross-communication), connections, variables and executors that allow parallel task execution.

From business requirements to working pipelines with apache airflowDerrick Qin

Apache Airflow at DailymotionGermain Tanguy

This document discusses Apache Airflow and its use at Dailymotion. It provides an agenda that covers data at Dailymotion, Apache Airflow, how Airflow is used at Dailymotion, deployment of Airflow at Dailymotion, working on a DAG (directed acyclic graph) pipeline, and an example pipeline for Dailymotion's new Advanced Analytics project. The example pipeline aggregates data from different sources with varying frequencies and timezones into BigQuery and Exasol for visualization in Tableau.

Airflow introductionChandler Huang

This document provides an overview of Apache Airflow, an open-source workflow management system. It describes Airflow's key features like workflow definition using directed acyclic graphs (DAGs), rich UI, scheduler, operators for tasks like databases and web services, and use of Jinja templating. The document also discusses Airflow's architecture with parallel execution, UI, command line operations like backfilling, and security features. Airflow is used by over 200 companies for workflows like ETL, analytics, and machine learning pipelines.

Apache AirflowKnoldus Inc.

Airflow presentationIlias Okacha

Airflow is a workflow management system for authoring, scheduling and monitoring workflows or directed acyclic graphs (DAGs) of tasks. It has features like DAGs to define tasks and their relationships, operators to describe tasks, sensors to monitor external systems, hooks to connect to external APIs and databases, and a user interface for visualizing pipelines and monitoring runs. Airflow uses a variety of executors like SequentialExecutor, CeleryExecutor and MesosExecutor to run tasks on schedulers like Celery or Kubernetes. It provides security features like authentication, authorization and impersonation to manage access.

Building an analytics workflow using Apache AirflowYohei Onishi

This document discusses using Apache Airflow to build an analytics workflow. It begins with an overview of Airflow and how it can be used to author workflows through Python code. Examples are shown of using Airflow to copy files between S3 buckets. The document then covers setting up a highly available Airflow cluster, implementing continuous integration/deployment, and monitoring workflows. It emphasizes that Google Cloud Composer can simplify deploying and managing Airflow clusters on Google Kubernetes Engine and integrating with other Google Cloud services.

Prometheus loves GrafanaTobias Schmidt

GrafanaCon 2015 - http://grafanacon.org/ Tobias will be giving an overview of Prometheus, an open-source monitoring system with a multi-dimensional label system, expressive query language and dashboard editor called PromDash. Learn about the highlights and differences of PromDash compared to Grafana and discuss the options to make Grafana the primary dashboard editor of the Prometheus project.

The GrapQL ecosystemOlegsGabrusjonoks

This document provides an overview of the GraphQL ecosystem, including popular client libraries, developer tools, GraphQL servers, and server libraries. It introduces Apollo Client and Relay as popular GraphQL client libraries for React, Angular, and Vue. Tools discussed include GraphiQL, Launchpad, Apollo Client Dev Tools, and schema visualizers. Express GraphQL, Apollo Server, and GraphQL Yoga are presented as common GraphQL server options for Node.js. Key server libraries that handle common problems like the N+1 queries issue, duplicate requests, and query complexity are also summarized.

Serverless in-actionAssaf Gannon

Serverless technologies and capabilities are here and are accessible now more than ever. The power of infinite scale and system capabilities has never been more accessible. This also affects traditional front end development as serverless technologies allow for easy construction of backend support for any frontend with ease and simplicity. In this talk, we will demonstrate how to build a fully functional Graphql endpoint for FE applications using Apollo Server and Client libraries, utilizing different cloud providers. We will also demonstrate the usage of Servless.com framework to set up the required infrastructure as code to simplify and support this setup The video of the presentation (Hebrew): https://youtu.be/8ba4cpdtK-8

Apache AirflowSumit Maheshwari

Apache airflowPurna Chander

This document provides an overview of building data pipelines using Apache Airflow. It discusses what a data pipeline is, common components of data pipelines like data ingestion and processing, and issues with traditional data flows. It then introduces Apache Airflow, describing its features like being fault tolerant and supporting Python code. The core components of Airflow including the web server, scheduler, executor, and worker processes are explained. Key concepts like DAGs, operators, tasks, and workflows are defined. Finally, it demonstrates Airflow through an example DAG that extracts and cleanses tweets.

It's a Breeze to develop Apache Airflow (London Apache Airflow meetup)Jarek Potiuk

This talk is about tools and mechanism we developed and used to improve productivity and teamwork in our team (of 6 currently) while developing 70+ operators for Airflow over more than 6 months. We developed an "Airflow Breeze" simplified development environment which cuts down the time to become productive Apache Airflow developer from days to minutes. It is part of Airflow Improvement Proposals: AIP-10 Multi-layered and multi-stage official Airflow image AIP-7 Simplified development workflow

Building a Data Pipeline using Apache Airflow (on AWS / GCP)Yohei Onishi

Influx/Days 2017 San Francisco | Dan Cech InfluxData

DATA VISUALIZATION & ALERTING WITH GRAFANA Grafana is the leading graph and dashboard builder for visualizing time series, which is a great tool for visual monitoring of InfluxData. This session will provide an intro to Grafana and talk about adding data sources, creating dashboards and getting the most out of your data visualization. The talk will look into some new features Grafana has to offer, as well as explain why different graphs are important and specifically how you can use them to analyze data performance and troubleshoot operational issues.

Orchestrating workflows Apache Airflow on GCP & AWSDerrick Qin

Working in a cloud or on-premises environment, we all somehow move data from A to B on-demand or on schedule. It is essential to have a tool that can automate recurring workflows. This can be anything from an ETL(Extract, Transform, and Load) job for a regular analytics report all the way to automatically re-training a machine learning model. In this talk, we will introduce Apache Airflow and how it can help orchestrate your workflows. We will cover key concepts, features, and use cases of Apache Airflow, as well as how you can enjoy Apache Airflow on GCP and AWS by demo-ing a few practical workflows.

InfluxDB and Grafana: An Introduction to Time-Based Data Storage and Visualiz...Caner Ünal

Contributing to Apache Airflow | Journey to becoming Airflow's leading contri...Kaxil Naik

From not knowing Python (let alone Airflow), and from submitting the first PR that fixes typo to becoming Airflow Committer, PMC Member, Release Manager, and #1 Committer this year, this talk walks through Kaxil’s journey in the Airflow World. The second part of this talk explains: how you can also start your OSS journey by contributing to Airflow Expanding familiarity with a different part of the Airflow codebase Continue committing regularly & steadily to become Airflow Committer. (including talking about current Guidelines of becoming a Committer) Different mediums of communication (Dev list, users list, Slack channel, Github Discussions etc)

Apache Airflow ArchitectureGerard Toonstra

Presentation given at Coolblue B.V. demonstrating Apache Airflow (incubating), what we learned from the underlying design principles and how an implementation of these principles reduce the amount of ETL effort. Why choose Airflow? Because it makes your engineering life easier, more people can contribute to how data flows through the organization, so that you can spend more time applying your brain to more difficult problems like Machine Learning, Deep Learning and higher level analysis.

Load test REST APIs using gatlingJayaram Sankaranarayanan

React meets o OCalmMichał Załęcki

Reason meets OCaml - Reason is a syntax extension to OCaml that compiles to efficient JavaScript. It has a rock solid type system and access to the OCaml ecosystem. - ReasonReact provides React bindings for Reason. Components are defined in a similar way to TypeScript but use Reason's type system and compiler. - Examples show how stateless and stateful components are defined in ReasonReact compared to TypeScript. ReasonReact uses OCaml's type system and reducer pattern while TypeScript uses interfaces and state. - Interoperability with JavaScript is supported through modules, raw JavaScript blocks, and using the platform. Reason aims to leverage the strengths of both OCaml and React.

OASGraph LoopBack 4 IntegrationMario Estrada

Apache Airflow IntroductionLiangjun Jiang

Apache Airflow is a platform for authoring, scheduling, and monitoring workflows or directed acyclic graphs (DAGs). It allows defining and monitoring cron jobs, automating DevOps tasks, moving data periodically, and building machine learning pipelines. Many large companies use Airflow for tasks like data ingestion, analytics automation, and machine learning workflows. The author proposes using Airflow to manage data movement and automate tasks for their organization to benefit business units. Instructions are provided on installing Airflow using pip, Docker, or Helm along with developing sample DAGs connecting to Azure services like Blob Storage, Cosmos DB, and Databricks.

Airflow tutorials hands_onpko89403

This document provides an overview of Airflow, an open-source workflow management platform for authoring, scheduling and monitoring data pipelines. It describes Airflow's key components including the web server, scheduler, workers and metadata database. It explains how Airflow works by parsing DAGs, instantiating tasks and changing their state as they are scheduled, queued, run and monitored. The document also covers concepts like DAGs, operators, dependencies, concurrency vs parallelism and advanced topics such as subDAGs, hooks, XCOM and branching workflows.

University program - writing an apache apex applicationAkshay Gore

What's coming in Airflow 2.0? - NYC Apache Airflow MeetupKaxil Naik

The document discusses upcoming features and changes in Apache Airflow 2.0. Key points include: 1. Scheduler high availability will use an active-active model with row-level locks to allow killing a scheduler without interrupting tasks. 2. DAG serialization will decouple DAG parsing from scheduling to reduce delays, support lazy loading, and enable features like versioning. 3. Performance improvements include optimizing the DAG file processor and using a profiling tool to identify other bottlenecks. 4. The Kubernetes executor will integrate with KEDA for autoscaling and allow customizing pods through templating. 5. The official Helm chart, functional DAGs, and smaller usability changes

Developing Commercial APEX ApplicationsEnkitec

This document discusses the development of a commercial APEX application called eSERT. Key points covered include designing the application for multiple APEX versions, using source control and automated builds, implementing product licensing, and deploying the application via a multi-part install process similar to APEX. Infrastructure choices like AWS, CodeSpaces, and virtual machines for shared development and testing are also summarized.

Building a Flexible UI with Oracle ApExBradley Brown

This document describes building a flexible user-definable application with Oracle Application Express (ApEx) to provide political intelligence to campaigns. Key points: 1) The application allows customization by changing data instead of code, making it flexible. Data such as attributes, people, and properties can be customized. 2) It uses Google Maps and provides visual searches to find political contributors matching filters. Materialized views improve performance. 3) The application was created for a company called Independent Nation but is designed to be generic for any customer needing mapping capabilities. User-specific data, filters, and default settings allow customization for each user.

More Related Content

What's hot (20)

Building an analytics workflow using Apache AirflowYohei Onishi

Prometheus loves GrafanaTobias Schmidt

The GrapQL ecosystemOlegsGabrusjonoks

Serverless in-actionAssaf Gannon

Apache AirflowSumit Maheshwari

Apache airflowPurna Chander

It's a Breeze to develop Apache Airflow (London Apache Airflow meetup)Jarek Potiuk

Building a Data Pipeline using Apache Airflow (on AWS / GCP)Yohei Onishi

Influx/Days 2017 San Francisco | Dan Cech InfluxData

Orchestrating workflows Apache Airflow on GCP & AWSDerrick Qin

InfluxDB and Grafana: An Introduction to Time-Based Data Storage and Visualiz...Caner Ünal

Contributing to Apache Airflow | Journey to becoming Airflow's leading contri...Kaxil Naik

Apache Airflow ArchitectureGerard Toonstra

Load test REST APIs using gatlingJayaram Sankaranarayanan

React meets o OCalmMichał Załęcki

OASGraph LoopBack 4 IntegrationMario Estrada

Apache Airflow IntroductionLiangjun Jiang

Airflow tutorials hands_onpko89403

University program - writing an apache apex applicationAkshay Gore

What's coming in Airflow 2.0? - NYC Apache Airflow MeetupKaxil Naik

Building an analytics workflow using Apache AirflowYohei Onishi

Prometheus loves GrafanaTobias Schmidt

The GrapQL ecosystemOlegsGabrusjonoks

Serverless in-actionAssaf Gannon

Apache AirflowSumit Maheshwari

Apache airflowPurna Chander

It's a Breeze to develop Apache Airflow (London Apache Airflow meetup)Jarek Potiuk

Building a Data Pipeline using Apache Airflow (on AWS / GCP)Yohei Onishi

Influx/Days 2017 San Francisco | Dan Cech InfluxData

Orchestrating workflows Apache Airflow on GCP & AWSDerrick Qin

InfluxDB and Grafana: An Introduction to Time-Based Data Storage and Visualiz...Caner Ünal

Contributing to Apache Airflow | Journey to becoming Airflow's leading contri...Kaxil Naik

Apache Airflow ArchitectureGerard Toonstra

Load test REST APIs using gatlingJayaram Sankaranarayanan

React meets o OCalmMichał Załęcki

OASGraph LoopBack 4 IntegrationMario Estrada

Apache Airflow IntroductionLiangjun Jiang

Airflow tutorials hands_onpko89403

University program - writing an apache apex applicationAkshay Gore

What's coming in Airflow 2.0? - NYC Apache Airflow MeetupKaxil Naik

Viewers also liked (7)

Developing Commercial APEX ApplicationsEnkitec

Building a Flexible UI with Oracle ApExBradley Brown

APEX 5 Demo and Best PracticesDimitri Gielis

Application softwareshalivale

Application software is designed to help users perform specific tasks. There are several types of application software including programming software, system software, word processing software, spreadsheet software, database software, presentation software, and multimedia software. Examples of application software include enterprise software, accounting software, office suites, and graphics software.

Application softwareRana Usman Sattar

Application software consists of programs designed to assist users with tasks like business activities, graphics projects, personal tasks, and communication. There are many types of application software including packaged software sold in stores, custom software for specific needs, web applications, open source software, and freeware or shareware distributed for free or for a trial period. Common business application software includes word processing, spreadsheet, database, presentation, and accounting programs to help users and businesses be more efficient.

Medical Store Management System Software Engineering Projecthani2253

This document provides an overview of a medical store management system project. It describes the project title, objectives, features, scope, and deliverables. The project aims to automate the inventory, accounting, and customer management processes of a medical store to ease the workload. It will use a waterfall model and be developed in Java. Key features will include product, customer, sales, and payment management. The document outlines requirements, design, and implementation plans including user stories, data flow diagrams, and a work breakdown structure.

Salesforce PresentationChetna Purohit

The document provides an overview of Salesforce development. It discusses that Salesforce is a cloud-based CRM platform with over 82,000 customers. It offers features for marketing, sales, services, and more. Salesforce can be customized through tools like Sales Cloud, Service Cloud, and Custom Cloud. Development in Salesforce involves creating objects and relationships, automating processes with workflows and approvals, building reports and dashboards, adding programming logic with Apex, and creating custom interfaces with Visualforce.

Developing Commercial APEX ApplicationsEnkitec

Building a Flexible UI with Oracle ApExBradley Brown

APEX 5 Demo and Best PracticesDimitri Gielis

Application softwareshalivale

Application softwareRana Usman Sattar

Medical Store Management System Software Engineering Projecthani2253

Salesforce PresentationChetna Purohit

Similar to Building your first aplication using Apache Apex (20)

Deep Dive into Apache Apex App DevelopmentApache Apex

Java High Level Stream APIApache Apex

Presenter - Siyuan Hua, Apache Apex PMC Member & DataTorrent Engineer Apache Apex provides a DAG construction API that gives the developers full control over the logical plan. Some use cases don't require all of that flexibility, at least so it may appear initially. Also a large part of the audience may be more familiar with an API that exhibits more functional programming flavor, such as the new Java 8 Stream interfaces and the Apache Flink and Spark-Streaming API. Thus, to make Apex beginners to get simple first app running with familiar API, we are now providing the Stream API on top of the existing DAG API. The Stream API is designed to be easy to use yet flexible to extend and compatible with the native Apex API. This means, developers can construct their application in a way similar to Flink, Spark but also have the power to fine tune the DAG at will. Per our roadmap, the Stream API will closely follow Apache Beam (aka Google Data Flow) model. In the future, you should be able to either easily run Beam applications with the Apex Engine or express an existing application in a more declarative style.

Apache Big Data EU 2016: Building Streaming Applications with Apache ApexApache Apex

Stream processing applications built on Apache Apex run on Hadoop clusters and typically power analytics use cases where availability, flexible scaling, high throughput, low latency and correctness are essential. These applications consume data from a variety of sources, including streaming sources like Apache Kafka, Kinesis or JMS, file based sources or databases. Processing results often need to be stored in external systems (sinks) for downstream consumers (pub-sub messaging, real-time visualization, Hive and other SQL databases etc.). Apex has the Malhar library with a wide range of connectors and other operators that are readily available to build applications. We will cover key characteristics like partitioning and processing guarantees, generic building blocks for new operators (write-ahead-log, incremental state saving, windowing etc.) and APIs for application specification.

Stream Processing use cases and applications with Apache Apex by Thomas WeiseBig Data Spain

Intro to Apache Apex @ Women in Big DataApache Apex

BigDataSpain 2016: Stream Processing Applications with Apache ApexThomas Weise

Developing streaming applications with apache apex (strata + hadoop world)Apache Apex

David Yan offers an overview of Apache Apex, a stream processing engine used in production by several large companies for real-time data analytics. Apache Apex uses a programming paradigm based on a directed acyclic graph (DAG). Each node in the DAG represents an operator, which can be data input, data output, or data transformation. Each directed edge in the DAG represents a stream, which is the flow of data from one operator to another. As part of Apex, the Malhar library provides a suite of connector operators so that Apex applications can read from or write to various data sources. It also includes utility operators that are commonly used in streaming applications, such as parsers, deduplicators and join, and generic building blocks that facilitate scalable state management and checkpointing. In addition to processing based on ingression time and processing time, Apex supports event-time windows and session windows. It also supports windowing, watermarks, allowed lateness, accumulation mode, triggering, and retraction detailed by Apache Beam as well as feedback loops in the DAG for iterative processing and at-least-once and “end-to-end” exactly-once processing guarantees. Apex provides various ways to fine-tune applications, such as operator partitioning, locality, and affinity. Apex is integrated with several open source projects, including Apache Beam, Apache Samoa (distributed machine learning), and Apache Calcite (SQL-based application specification). Users can choose Apex as the backend engine when running their application model based on these projects. David explains how to develop fault-tolerant streaming applications with low latency and high throughput using Apex, presenting the programming model with examples and demonstrating how custom business logic can be integrated using both the declarative high-level API and the compositional DAG-level API.

Apache Apex Introduction with PubMaticApache Apex

Introduction to Apache Apex and writing a big data streaming application Apache Apex

Introduction to Apache Apex - The next generation native Hadoop platform, and writing a native Hadoop big data Apache Apex streaming application. This talk will cover details about how Apex can be used as a powerful and versatile platform for big data. Apache apex is being used in production by customers for both streaming and batch use cases. Common usage of Apache Apex includes big data ingestion, streaming analytics, ETL, fast batch. alerts, real-time actions, threat detection, etc. Presenter : <b>Pramod Immaneni</b> Apache Apex PPMC member and senior architect at DataTorrent Inc, where he works on Apex and specializes in big data applications. Prior to DataTorrent he was a co-founder and CTO of Leaf Networks LLC, eventually acquired by Netgear Inc, where he built products in core networking space and was granted patents in peer-to-peer VPNs. Before that he was a technical co-founder of a mobile startup where he was an architect of a dynamic content rendering engine for mobile devices. This is a video of the webcast of an Apache Apex meetup event organized by Guru Virtues at 267 Boston Rd no. 9, North Billerica, MA, on <b>May 7th 2016</b> and broadcasted from San Jose, CA. If you are interested in helping organize i.e., hosting, presenting, community leadership Apache Apex community, please email [email protected]

Introduction to Apache ApexChinmay Kolhatkar

Intro to Apache Apex - Next Gen Native Hadoop Platform - HackacApache Apex

Apache Apex is a platform and runtime engine that enables development of scalable and fault-tolerant distributed applications on Hadoop in a native fashion. It processes streaming or batch big data with high throughput and low latency. Applications are built from operators that run distributed across a cluster and can scale up or down dynamically. Apex provides automatic recovery from failures without reprocessing and preserves state. It includes a library of common operators to simplify application development.

Introduction to Apache ApexApache Apex

Real-time Stream Processing using Apache ApexApache Apex

Apache Apex is a stream processing framework that provides high performance, scalability, and fault tolerance. It uses YARN for resource management, can achieve single digit millisecond latency, and automatically recovers from failures without data loss through checkpointing. Apex applications are modeled as directed acyclic graphs of operators and can be partitioned for scalability. It has a large community of committers and is in the process of becoming a top-level Apache project.

Introduction to Apache Apex - CoDS 2016Bhupesh Chawda

Intro to Apache Apex (next gen Hadoop) & comparison to Spark StreamingApache Apex

Presenter: Devendra Tagare - DataTorrent Engineer, Contributor to Apex, Data Architect experienced in building high scalability big data platforms. Apache Apex is a next generation native Hadoop big data platform. This talk will cover details about how it can be used as a powerful and versatile platform for big data. Apache Apex is a native Hadoop data-in-motion platform. We will discuss architectural differences between Apache Apex features with Spark Streaming. We will discuss how these differences effect use cases like ingestion, fast real-time analytics, data movement, ETL, fast batch, very low latency SLA, high throughput and large scale ingestion. We will cover fault tolerance, low latency, connectors to sources/destinations, smart partitioning, processing guarantees, computation and scheduling model, state management and dynamic changes. We will also discuss how these features affect time to market and total cost of ownership.

Introduction to Apache Apex by Thomas WeiseBig Data Spain

Actionable Insights with Apache Apex at Apache Big Data 2017 by Devendra TagareApache Apex

The presentation covers how Apache Apex is used to deliver actionable insights in real-time for Ad-tech. It includes a reference architecture to provide dimensional aggregates on TB scale for billions of events per day. The reference architecture covers concepts around Apache Apex, with Kafka as source and dimensional compute. Slides from Devendra Tagare at Apache Big Data North America in Miami 2017.

Intro to Apache Apex - Next Gen Platform for Ingest and TransformApache Apex

Introduction to Apache Apex - The next generation native Hadoop platform. This talk will cover details about how Apache Apex can be used as a powerful and versatile platform for big data processing. Common usage of Apache Apex includes big data ingestion, streaming analytics, ETL, fast batch alerts, real-time actions, threat detection, etc. Bio: Pramod Immaneni is Apache Apex PMC member and senior architect at DataTorrent, where he works on Apache Apex and specializes in big data platform and applications. Prior to DataTorrent, he was a co-founder and CTO of Leaf Networks LLC, eventually acquired by Netgear Inc, where he built products in core networking space and was granted patents in peer-to-peer VPNs.

Apache Big Data EU 2016: Next Gen Big Data Analytics with Apache ApexApache Apex

Stream data processing is becoming increasingly important to support business needs for faster time to insight and action with growing volume of information from more sources. Apache Apex (http://apex.apache.org/) is a unified big data in motion processing platform for the Apache Hadoop ecosystem. Apex supports demanding use cases with: * Architecture for high throughput, low latency and exactly-once processing semantics. * Comprehensive library of building blocks including connectors for Kafka, Files, Cassandra, HBase and many more * Java based with unobtrusive API to build real-time and batch applications and implement custom business logic. * Advanced engine features for auto-scaling, dynamic changes, compute locality. Apex was developed since 2012 and is used in production in various industries like online advertising, Internet of Things (IoT) and financial services.

BigDataSpain 2016: Introduction to Apache ApexThomas Weise

Apache Apex is an open source stream processing platform, built for large scale, high-throughput, low-latency, high availability and operability. With a unified architecture it can be used for real-time and batch processing. Apex is Java based and runs natively on Apache Hadoop YARN and HDFS. We will discuss the key features of Apache Apex and architectural differences from similar platforms and how these differences affect use cases like ingestion, fast real-time analytics, data movement, ETL, fast batch, low latency SLA, high throughput and large scale ingestion. Apex APIs and libraries of operators and examples focus on developer productivity. We will present the programming model with examples and how custom business logic can be easily integrated based on the Apex operator API. We will cover integration with connectors to sources/destinations (including Kafka, JMS, SQL, NoSQL, files etc.), scalability with advanced partitioning, fault tolerance and processing guarantees, computation and scheduling model, state management, windowing and dynamic changes. Attendees will also learn how these features affect time to market and total cost of ownership and how they are important in existing Apex production deployments. https://www.bigdataspain.org/

Deep Dive into Apache Apex App DevelopmentApache Apex

Java High Level Stream APIApache Apex

Apache Big Data EU 2016: Building Streaming Applications with Apache ApexApache Apex

Stream Processing use cases and applications with Apache Apex by Thomas WeiseBig Data Spain

Intro to Apache Apex @ Women in Big DataApache Apex

BigDataSpain 2016: Stream Processing Applications with Apache ApexThomas Weise

Developing streaming applications with apache apex (strata + hadoop world)Apache Apex

Apache Apex Introduction with PubMaticApache Apex

Introduction to Apache Apex and writing a big data streaming application Apache Apex

Introduction to Apache ApexChinmay Kolhatkar

Intro to Apache Apex - Next Gen Native Hadoop Platform - HackacApache Apex

Introduction to Apache ApexApache Apex

Real-time Stream Processing using Apache ApexApache Apex

Introduction to Apache Apex - CoDS 2016Bhupesh Chawda

Intro to Apache Apex (next gen Hadoop) & comparison to Spark StreamingApache Apex

Introduction to Apache Apex by Thomas WeiseBig Data Spain

Actionable Insights with Apache Apex at Apache Big Data 2017 by Devendra TagareApache Apex

Intro to Apache Apex - Next Gen Platform for Ingest and TransformApache Apex

Apache Big Data EU 2016: Next Gen Big Data Analytics with Apache ApexApache Apex

BigDataSpain 2016: Introduction to Apache ApexThomas Weise

Recently uploaded (20)

How to Detect Outliers in IBM SPSS Statistics.pptxVersion 1 Analytics

Oracle Cloud Infrastructure AI FoundationsVICTOR MAESTRE RAMIREZ

Mastering AI Workflows with FME - Peak of Data & AI 2025Safe Software

Harness the full potential of AI with FME: From creating high-quality training data to optimizing models and utilizing results, FME supports every step of your AI workflow. Seamlessly integrate a wide range of models, including those for data enhancement, forecasting, image and object recognition, and large language models. Customize AI models to meet your exact needs with FME’s powerful tools for training, optimization, and seamless integration

LSNIF: Locally-Subdivided Neural Intersection FunctionTakahiro Harada

Neural representations have shown the potential to accelerate ray casting in a conventional ray-tracing-based rendering pipeline. We introduce a novel approach called Locally-Subdivided Neural Intersection Function (LSNIF) that replaces bottom-level BVHs used as traditional geometric representations with a neural network. Our method introduces a sparse hash grid encoding scheme incorporating geometry voxelization, a scene-agnostic training data collection, and a tailored loss function. It enables the network to output not only visibility but also hit-point information and material indices. LSNIF can be trained offline for a single object, allowing us to use LSNIF as a replacement for its corresponding BVH. With these designs, the network can handle hit-point queries from any arbitrary viewpoint, supporting all types of rays in the rendering pipeline. We demonstrate that LSNIF can render a variety of scenes, including real-world scenes designed for other path tracers, while achieving a memory footprint reduction of up to 106.2x compared to a compressed BVH. https://arxiv.org/abs/2504.21627

Dancing with AI - A Developer's Journey.pptxElliott Richmond

In this talk, Elliott explores how developers can embrace AI not as a threat, but as a collaborative partner. We’ll examine the shift from routine coding to creative leadership, highlighting the new developer superpowers of vision, integration, and innovation. We'll touch on security, legacy code, and the future of democratized development. Whether you're AI-curious or already a prompt engineering, this session will help you find your rhythm in the new dance of modern development.

Scaling GenAI Inference From Prototype to Production: Real-World Lessons in S...Anish Kumar

Presented by: Anish Kumar LinkedIn: https://www.linkedin.com/in/anishkumar/ This lightning talk dives into real-world GenAI projects that scaled from prototype to production using Databricks’ fully managed tools. Facing cost and time constraints, we leveraged four key Databricks features—Workflows, Model Serving, Serverless Compute, and Notebooks—to build an AI inference pipeline processing millions of documents (text and audiobooks). This approach enables rapid experimentation, easy tuning of GenAI prompts and compute settings, seamless data iteration and efficient quality testing—allowing Data Scientists and Engineers to collaborate effectively. Learn how to design modular, parameterized notebooks that run concurrently, manage dependencies and accelerate AI-driven insights. Whether you're optimizing AI inference, automating complex data workflows or architecting next-gen serverless AI systems, this session delivers actionable strategies to maximize performance while keeping costs low.

Boosting MySQL with Vector Search -THE VECTOR SEARCH CONFERENCE 2025 .pdfAlkin Tezuysal

As the demand for vector databases and Generative AI continues to rise, integrating vector storage and search capabilities into traditional databases has become increasingly important. This session introduces the *MyVector Plugin*, a project that brings native vector storage and similarity search to MySQL. Unlike PostgreSQL, which offers interfaces for adding new data types and index methods, MySQL lacks such extensibility. However, by utilizing MySQL's server component plugin and UDF, the *MyVector Plugin* successfully adds a fully functional vector search feature within the existing MySQL + InnoDB infrastructure, eliminating the need for a separate vector database. The session explains the technical aspects of integrating vector support into MySQL, the challenges posed by its architecture, and real-world use cases that showcase the advantages of combining vector search with MySQL's robust features. Attendees will leave with practical insights on how to add vector search capabilities to their MySQL systems.

“State-space Models vs. Transformers for Ultra-low-power Edge AI,” a Presenta...Edge AI and Vision Alliance

For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2025/06/state-space-models-vs-transformers-for-ultra-low-power-edge-ai-a-presentation-from-brainchip/ Tony Lewis, Chief Technology Officer at BrainChip, presents the “State-space Models vs. Transformers for Ultra-low-power Edge AI” tutorial at the May 2025 Embedded Vision Summit. At the embedded edge, choices of language model architectures have profound implications on the ability to meet demanding performance, latency and energy efficiency requirements. In this presentation, Lewis contrasts state-space models (SSMs) with transformers for use in this constrained regime. While transformers rely on a read-write key-value cache, SSMs can be constructed as read-only architectures, enabling the use of novel memory types and reducing power consumption. Furthermore, SSMs require significantly fewer multiply-accumulate units—drastically reducing compute energy and chip area. New techniques enable distillation-based migration from transformer models such as Llama to SSMs without major performance loss. In latency-sensitive applications, techniques such as precomputing input sequences allow SSMs to achieve sub-100 ms time-to-first-token, enabling real-time interactivity. Lewis presents a detailed side-by-side comparison of these architectures, outlining their trade-offs and opportunities at the extreme edge.

Domino IQ – Was Sie erwartet, erste Schritte und Anwendungsfällepanagenda

Webinar Recording: https://www.panagenda.com/webinars/domino-iq-was-sie-erwartet-erste-schritte-und-anwendungsfalle/ HCL Domino iQ Server – Vom Ideenportal zur implementierten Funktion. Entdecken Sie, was es ist, was es nicht ist, und erkunden Sie die Chancen und Herausforderungen, die es bietet. Wichtige Erkenntnisse - Was sind Large Language Models (LLMs) und wie stehen sie im Zusammenhang mit Domino iQ - Wesentliche Voraussetzungen für die Bereitstellung des Domino iQ Servers - Schritt-für-Schritt-Anleitung zur Einrichtung Ihres Domino iQ Servers - Teilen und diskutieren Sie Gedanken und Ideen, um das Potenzial von Domino iQ zu maximieren

MCP vs A2A vs ACP: Choosing the Right Protocol | BluebashBluebash

End-to-end Assurance for SD-WAN & SASE with ThousandEyesThousandEyes

Data Virtualization: Bringing the Power of FME to Any ApplicationSafe Software

Imagine building web applications or dashboards on top of all your systems. With FME’s new Data Virtualization feature, you can deliver the full CRUD (create, read, update, and delete) capabilities on top of all your data that exploit the full power of FME’s all data, any AI capabilities. Data Virtualization enables you to build OpenAPI compliant API endpoints using FME Form’s no-code development platform. In this webinar, you’ll see how easy it is to turn complex data into real-time, usable REST API based services. We’ll walk through a real example of building a map-based app using FME’s Data Virtualization, and show you how to get started in your own environment – no dev team required. What you’ll take away: -How to build live applications and dashboards with federated data -Ways to control what’s exposed: filter, transform, and secure responses -How to scale access with caching, asynchronous web call support, with API endpoint level security. -Where this fits in your stack: from web apps, to AI, to automation Whether you’re building internal tools, public portals, or powering automation – this webinar is your starting point to real-time data delivery.

Domino IQ – What to Expect, First Steps and Use Casespanagenda

Webinar Recording: https://www.panagenda.com/webinars/domino-iq-what-to-expect-first-steps-and-use-cases/ HCL Domino iQ Server – From Ideas Portal to implemented Feature. Discover what it is, what it isn’t, and explore the opportunities and challenges it presents. Key Takeaways - What are Large Language Models (LLMs) and how do they relate to Domino iQ - Essential prerequisites for deploying Domino iQ Server - Step-by-step instructions on setting up your Domino iQ Server - Share and discuss thoughts and ideas to maximize the potential of Domino iQ

Agentic AI: Beyond the Buzz- LangGraph Studio V2Shashikant Jagtap

ELNL2025 - Unlocking the Power of Sensitivity Labels - A Comprehensive Guide....Jasper Oosterveld

Sensitivity labels, powered by Microsoft Purview Information Protection, serve as the foundation for classifying and protecting your sensitive data within Microsoft 365. Their importance extends beyond classification and play a crucial role in enforcing governance policies across your Microsoft 365 environment. Join me, a Data Security Consultant and Microsoft MVP, as I share practical tips and tricks to get the full potential of sensitivity labels. I discuss sensitive information types, automatic labeling, and seamless integration with Data Loss Prevention, Teams Premium, and Microsoft 365 Copilot.

cnc-drilling-dowel-inserting-machine-drillteq-d-510-english.pdfAmirStern2

7 Salesforce Data Cloud Best Practices.pdfMinuscule Technologies

6th Power Grid Model Meetup - 21 May 2025DanBrown980551

6th Power Grid Model Meetup Join the Power Grid Model community for an exciting day of sharing experiences, learning from each other, planning, and collaborating. This hybrid in-person/online event will include a full day agenda, with the opportunity to socialize afterwards for in-person attendees. If you have a hackathon proposal, tell us when you register! About Power Grid Model The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services. Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability. Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.

If You Use Databricks, You Definitely Need FMESafe Software

DevOps in the Modern Era - Thoughtfully Critical PodcastChris Wahl