Scaling Analytics with elasticsearch
Download as pptx, pdf18 likes10,211 views
This document summarizes how Elasticsearch can be used for scaling analytics applications. Elasticsearch is an open source, distributed search and analytics engine that can index large volumes of data. It automatically shards and replicates data across nodes for redundancy and high availability. Analytics queries like date histograms, statistical facets, and geospatial searches can retrieve insightful results from large datasets very quickly. The document provides an example of using Elasticsearch to perform sentiment analysis, location tagging, and analytical queries on over 100 million social media documents.
1 of 28
Downloaded 104 times
![Sample Document
es.post('articles/facebook', data={
”date": "2012-09-01 08:37:55",
"tags": {
"sentiment": {
"positive": 0.36,
"negative": 0.10
}
"geotags": [{
"term" : "Cairo",
"location" : "30.0566,31.2262”,
“type” : “geo_point”
}],
"search_terms": [
"Mohamed Morsi"
]
},
"item": {
"publisher: "Facebook"
"source_domain": "www.facebook.com",
"author": "James Smith",
"source_url": "http://www.facebook.com/5551231234/posts/414141414141",
"content_text": "Mohamed Morsi visits Iran for first time since 1979 ....",
"title": "James Smith posted a note to Facebook",
"author_url: "http://www.facebook.com/profile.php?id=5551231234"
}
})](https://image.slidesharecdn.com/scalinganalyticswithelasticsearch-120913214509-phpapp01/85/Scaling-Analytics-with-elasticsearch-19-320.jpg)
![Statistical Facet for Sentiment: Result
{
"facets": {
"sentiment_stats": {
"_type": "statistical",
"count": 8825,
"max": 0.375,
"mean": 0.008503991588291782,
"min": 0.0,
"std_deviation": 0.021251077265305472,
"sum_of_squares": 4.623648343200283,
"total": 75.04772576667497,
"variance": 0.00045160828493598306
}
},
"hits": {
"hits": [],
"max_score": 1.1120162,
"total": 8825
},
"took": 60
}](https://image.slidesharecdn.com/scalinganalyticswithelasticsearch-120913214509-phpapp01/85/Scaling-Analytics-with-elasticsearch-24-320.jpg)
Ad
Recommended
Using elasticsearch with rails
Using elasticsearch with railsTom Z Zeng Elasticsearch is a powerful open source search and analytics engine. It allows for full text search capabilities as well as powerful analytics functions. Elasticsearch can be used as both a search engine and as a NoSQL data store. It is easy to set up, use, scale, and maintain. The document provides examples of using Elasticsearch with Rails applications and discusses advanced features such as fuzzy search, autocomplete, and geospatial search.
Elasticsearch first-steps
Elasticsearch first-stepsMatteo Moci This document discusses using Elasticsearch for social media analytics and provides examples of common tasks. It introduces Elasticsearch basics like installation, indexing documents, and searching. It also covers more advanced topics like mapping types, facets for aggregations, analyzers, nested and parent/child relations between documents. The document concludes with recommendations on data design, suggesting indexing strategies for different use cases like per user, single index, or partitioning by time range.
ElasticSearch - index server used as a document database
ElasticSearch - index server used as a document databaseRobert Lujo The document provides an overview of Elasticsearch, a distributed, schema-free search server based on Apache Lucene, highlighting its features as a document database and its integration in the ELK stack. It emphasizes the use of Elasticsearch for real-time data analysis, multi-tenancy, and full-text search, while also contrasting its functionality with traditional relational databases. Additionally, it discusses hybrid solutions combining Elasticsearch with other databases, use cases, and advantages in document-oriented applications.
Elasticsearch
ElasticsearchRicardo Peres Natixis Open Day 2018 presentation about Elasticsearch:
- Elasticsearch is a distributed, RESTful search and analytics engine for indexing and searching JSON documents.
- It allows for distributed logging, document indexing, inexact searches, and custom relevance scoring.
- Documents are organized into indexes, types, and shards for distributed querying and storage.
- Documents can be created, updated, and deleted via REST API calls. Relevance can be customized through boosting, functions, and other scoring methods.
- Kibana provides visualization and analytics capabilities for Elasticsearch data. Logstash and Beats facilitate data collection and shipping.
Elasticsearch in 15 minutes
Elasticsearch in 15 minutesDavid Pilato This document provides a comprehensive overview of Elasticsearch, starting from its installation and basic usage to advanced features such as query DSL, document indexing, and real-time filtering. It includes practical examples of commands for creating, searching, and analyzing documents, as well as discussing facets and statistical aggregations. The document concludes with suggestions for additional features like bulk operations, percolators, and monitoring APIs.
Elasticsearch - DevNexus 2015
Elasticsearch - DevNexus 2015Roy Russo This document provides an introduction to Elasticsearch. It begins by introducing the speaker and their background. It then discusses what search is and how search engines work by using an inverted index to map tokens to documents. Elasticsearch is introduced as a search and analytics engine that is document-oriented, distributed, schema-free, and uses HTTP and JSON. It can be used for real-time search and analytics. The document discusses how Elasticsearch is based on Apache Lucene and can be run on multiple nodes in a cluster for high availability. It provides examples of using Elasticsearch for centralized logging, and discusses indexing, querying, and interacting with Elasticsearch via its RESTful API.
Intro to elasticsearch
Intro to elasticsearchJoey Wen 1) The document discusses information retrieval and search engines. It describes how search engines work by indexing documents, building inverted indexes, and allowing users to search indexed terms.
2) It then focuses on Elasticsearch, describing it as a distributed, open source search and analytics engine that allows for real-time search, analytics, and storage of schema-free JSON documents.
3) The key concepts of Elasticsearch include clusters, nodes, indexes, types, shards, and documents. Clusters hold the data and provide search capabilities across nodes.
Simple search with elastic search
Simple search with elastic searchmarkstory Elasticsearch is a JSON document database that allows for powerful full-text search capabilities. It uses Lucene under the hood for indexing and search. Documents are stored in indexes and types which are analogous to tables in a relational database. Documents can be created, read, updated, and deleted via a RESTful API. Searches can be performed across multiple indexes and types. Elasticsearch offers advanced search features like facets, highlighting, and custom analyzers. Mappings allow for customization of how documents are indexed. Shards and replicas improve performance and availability. Multi-tenancy can be achieved through separate indexes or filters.
Elasticsearch From the Bottom Up
Elasticsearch From the Bottom Upfoundsearch The document discusses Elasticsearch, its functionalities, and challenges faced during implementation. It highlights issues such as search query matching, index performance, and memory usage while explaining core concepts like segments and shards. The author, Njal Karevoll, emphasizes practical insights gained from years of experience managing Elasticsearch clusters.
Elasticsearch
ElasticsearchRicardo Peres This document provides an overview of Elasticsearch, including:
- It is a NoSQL database that indexes and searches JSON documents in real-time. Documents are distributed across a cluster of servers for high performance and availability.
- Elasticsearch uses Lucene under the hood for indexing and search. It is part of the ELK (Elasticsearch, Logstash, Kibana) stack and is open source.
- Documents are organized into indexes and types, similar to databases and tables. Documents can be created, updated, and deleted via a RESTful API.
ElasticSearch - DevNexus Atlanta - 2014
ElasticSearch - DevNexus Atlanta - 2014Roy Russo This document provides an introduction to Elasticsearch, including:
- Elasticsearch is a document-oriented search engine that uses JSON documents and Apache Lucene for indexing and searching. It allows for horizontal scaling across distributed nodes.
- Elasticsearch is well-suited for full-text search, acting as a fast read database with simple data structures to minimize impedance mismatch. Common uses include logs with Logstash, Elasticsearch, and Kibana.
- Key concepts include nodes, clusters, indexes, mapping types, shards, the inverted index, and the REST API for interacting with Elasticsearch.
Elasticsearch & "PeopleSearch"
Elasticsearch & "PeopleSearch"George Stathis The document discusses the use of Elasticsearch at Traackr for building a people discovery engine, migrating from Solr in 2012. It covers Elasticsearch's features such as near real-time search, automatic schema detection, and document types, while also addressing search performance challenges and alternatives like Solr grouping and denormalization. The presentation includes examples of search queries, schema mapping, and the advantages of using Elasticsearch over other solutions for real-time search needs.
Elasticsearch Introduction at BigData meetup
Elasticsearch Introduction at BigData meetupEric Rodriguez (Hiring in Lex) The document provides an introduction to Elasticsearch, a distributed search and analytics engine built on Apache Lucene. It outlines its architecture, core features, key components including Logstash and Kibana, and various use cases such as full-text search and data storage. Additionally, it discusses Elasticsearch's capabilities in handling high availability through sharding and replication, along with its extensive ecosystem of integrations and clients.
Solr vs. Elasticsearch - Case by Case
Solr vs. Elasticsearch - Case by CaseAlexandre Rafalovitch The document compares Solr and Elasticsearch, highlighting their similarities and differences, including aspects like full-text search capabilities, configuration methods, and plugin usage. It discusses specific technical details about their setups, indexing processes, and search functionalities, illustrating the strengths and weaknesses of both systems. Additionally, it covers topics such as nested documents, cloud deployment strategies, and explicit versus dynamic mapping in each tool.
Elasticsearch Introduction
Elasticsearch IntroductionRoopendra Vishwakarma Elasticsearch is a distributed, open source search and analytics engine built on Apache Lucene. It allows storing and searching of documents of any schema in JSON format. Documents are organized into indexes which can have multiple shards and replicas for scalability and high availability. Elasticsearch provides a RESTful API and can be easily extended with plugins. It is widely used for full-text search, structured search, analytics and more in applications requiring real-time search and analytics of large volumes of data.
Elastic search apache_solr
Elastic search apache_solrmacrochen Elasticsearch offers several advantages over Apache Solr including being more easily distributed, replicated, and supporting real-time indexing. It allows for easy sharding and replication of indexes across multiple nodes. However, Elasticsearch lacks some features found in Solr such as spell checking, date math, and facet pagination. The document provides an overview of the similarities and differences between Elasticsearch and Solr for choosing between the two search servers.
An Introduction to Elastic Search.
An Introduction to Elastic Search.Jurriaan Persyn The document discusses technical details of implementing full-text search using PHP/MySQL and highlights advanced features of Elasticsearch, a distributed search engine built on Apache Lucene. It covers aspects such as data indexing, tokenization, stemming, filtering, and relevance scoring. Additionally, it compares Elasticsearch with other search engines like Solr and Sphinx, and outlines the benefits of using Elasticsearch for efficient and scalable search solutions.
Elasticsearch as a search alternative to a relational database
Elasticsearch as a search alternative to a relational databaseKristijan Duvnjak The document provides an introduction to Elasticsearch, highlighting its features as a distributed, schema-free search engine built on Apache Lucene, suitable for users transitioning from relational databases. It details the structure of Elasticsearch clusters, indexing, shards, and the importance of JSON-based documents, as well as describing how data retrieval and search functionalities work. Additionally, it outlines practical use cases and performance benefits of Elasticsearch in comparison to traditional databases.
Elasticsearch for Data Analytics
Elasticsearch for Data AnalyticsFelipe This document provides an overview of using Elasticsearch for data analytics. It discusses various aggregation techniques in Elasticsearch like terms, min/max/avg/sum, cardinality, histogram, date_histogram, and nested aggregations. It also covers mappings, dynamic templates, and general tips for working with aggregations. The main takeaways are that aggregations in Elasticsearch provide insights into data distributions and relationships similarly to GROUP BY in SQL, and that mappings and templates can optimize how data is indexed for aggregation purposes.
Introduction to Elasticsearch
Introduction to ElasticsearchSperasoft The document provides an overview of Elasticsearch including that it is easy to install, horizontally scalable, and highly available. It discusses Elasticsearch's core search capabilities using Lucene and how data can be stored and retrieved. The document also covers Elasticsearch's distributed nature, plugins, scripts, custom analyzers, and other features like aggregations, filtering and sorting.
Introduction to Elasticsearch
Introduction to ElasticsearchJason Austin Elasticsearch is a distributed, RESTful search and analytics engine that allows for fast searching, filtering, and analysis of large volumes of data. It is document-based and stores structured and unstructured data in JSON documents within configurable indices. Documents can be queried using a simple query string syntax or more complex queries using the domain-specific query language. Elasticsearch also supports analytics through aggregations that can perform metrics and bucketing operations on document fields.
Workshop: Learning Elasticsearch
Workshop: Learning ElasticsearchAnurag Patel Elasticsearch is a distributed, open source search and analytics engine. It allows storing and searching of documents of any schema in real-time. Documents are organized into indices which can contain multiple types of documents. Indices are partitioned into shards and replicas to allow horizontal scaling and high availability. The document consists of a JSON object which is indexed and can be queried using a RESTful API.
ElasticSearch: Distributed Multitenant NoSQL Datastore and Search Engine
ElasticSearch: Distributed Multitenant NoSQL Datastore and Search EngineDaniel N The document provides an overview of a distributed multitenant NoSQL datastore and search engine, emphasizing concepts such as SQL vs NoSQL, the architecture of Elasticsearch, and core components like indices, shards, and nodes. It highlights the capabilities of Elasticsearch, including cloud compatibility, RESTful API support, and its scalability options. Additionally, it discusses practical usage examples, typical queries, and integrations with AWS services.
Intro to Elasticsearch
Intro to ElasticsearchClifford James Elasticsearch is an open-source, distributed, real-time document indexer with support for online analytics. It has features like a powerful REST API, schema-less data model, full distribution and high availability, and advanced search capabilities. Documents are indexed into indexes which contain mappings and types. Queries retrieve matching documents from indexes. Analysis converts text into searchable terms using tokenizers, filters, and analyzers. Documents are distributed across shards and replicas for scalability and fault tolerance. The REST APIs can be used to index, search, and inspect the cluster.
Elasticsearch: You know, for search! and more!
Elasticsearch: You know, for search! and more!Philips Kokoh Prasetyo Elasticsearch is presented as an expert in real-time search, aggregation, and analytics. The document outlines Elasticsearch concepts like indexing, mapping, analysis, and the query DSL. Examples are provided for real-time search queries, aggregations including terms, date histograms, and geo distance. Lessons learned from using Elasticsearch at LARC are also discussed.
Introduction to Elasticsearch
Introduction to ElasticsearchRuslan Zavacky Elasticsearch is a distributed, schema-free document store designed for real-time data indexing and analytics, built on Apache Lucene. It provides full-text search capabilities, supports JSON documents, and features a RESTful API for operations, ensuring data safety and quick access. The platform is open-source under the Apache 2.0 license, making it highly extensible and accessible for various applications.
Elastic search
Elastic searchNexThoughts Technologies Elasticsearch is an open-source search engine and analytics engine built on Apache Lucene that allows for real-time distributed search across indexes and analytics capabilities. It consists of clusters of nodes that store indexed data and can search across the clusters. The data is divided into shards and replicas can be made of shards for redundancy. Elasticsearch supports different analyzers for tokenizing text and filtering searches.
Presentation
Presentationkiarash1361 The document proposes a design for a durable, low-latency key-value data store. It discusses approaches to achieving durability like periodic snapshots and log replication. The proposed solution uses asynchronous log replication to remote storage units for durability and availability. The design uses chain replication of logs for synchronous writes and allows forming stable storage units across multiple machines for scalability. Evaluation shows the design can achieve high throughput and low latency even with additional replication.
Présentation de la solution Strategeex
Présentation de la solution StrategeexVisiativ Le document présente une solution logicielle, Strategeex, dédiée à l'analyse stratégique des organisations, permettant de formaliser des scénarios de développement et de piloter leur mise en œuvre. Il décrit également le processus d'analyse interne et externe, ainsi que la détermination des domaines d'activités stratégiques et l'évaluation de la position concurrentielle. Enfin, Strategeex facilite la formulation du diagnostic stratégique et la construction de plans d'amélioration à travers des outils de reporting et de pilotage adaptés.
ElasticSearch in action
ElasticSearch in actionCodemotion Elasticsearch is a full-text search engine and NoSQL analytics database built on Java and Lucene that supports real-time data processing and schema-less indexing. It offers RESTful APIs and features like inverted indices for enhanced search performance, and supports a variety of query types and mapping options for user-defined schemas. The document also covers specific operations, such as data insertion, mapping, and examples of various search and aggregation queries.
More Related Content
What's hot (19)
Elasticsearch From the Bottom Up
Elasticsearch From the Bottom Upfoundsearch The document discusses Elasticsearch, its functionalities, and challenges faced during implementation. It highlights issues such as search query matching, index performance, and memory usage while explaining core concepts like segments and shards. The author, Njal Karevoll, emphasizes practical insights gained from years of experience managing Elasticsearch clusters.
Elasticsearch
ElasticsearchRicardo Peres This document provides an overview of Elasticsearch, including:
- It is a NoSQL database that indexes and searches JSON documents in real-time. Documents are distributed across a cluster of servers for high performance and availability.
- Elasticsearch uses Lucene under the hood for indexing and search. It is part of the ELK (Elasticsearch, Logstash, Kibana) stack and is open source.
- Documents are organized into indexes and types, similar to databases and tables. Documents can be created, updated, and deleted via a RESTful API.
ElasticSearch - DevNexus Atlanta - 2014
ElasticSearch - DevNexus Atlanta - 2014Roy Russo This document provides an introduction to Elasticsearch, including:
- Elasticsearch is a document-oriented search engine that uses JSON documents and Apache Lucene for indexing and searching. It allows for horizontal scaling across distributed nodes.
- Elasticsearch is well-suited for full-text search, acting as a fast read database with simple data structures to minimize impedance mismatch. Common uses include logs with Logstash, Elasticsearch, and Kibana.
- Key concepts include nodes, clusters, indexes, mapping types, shards, the inverted index, and the REST API for interacting with Elasticsearch.
Elasticsearch & "PeopleSearch"
Elasticsearch & "PeopleSearch"George Stathis The document discusses the use of Elasticsearch at Traackr for building a people discovery engine, migrating from Solr in 2012. It covers Elasticsearch's features such as near real-time search, automatic schema detection, and document types, while also addressing search performance challenges and alternatives like Solr grouping and denormalization. The presentation includes examples of search queries, schema mapping, and the advantages of using Elasticsearch over other solutions for real-time search needs.
Elasticsearch Introduction at BigData meetup
Elasticsearch Introduction at BigData meetupEric Rodriguez (Hiring in Lex) The document provides an introduction to Elasticsearch, a distributed search and analytics engine built on Apache Lucene. It outlines its architecture, core features, key components including Logstash and Kibana, and various use cases such as full-text search and data storage. Additionally, it discusses Elasticsearch's capabilities in handling high availability through sharding and replication, along with its extensive ecosystem of integrations and clients.
Solr vs. Elasticsearch - Case by Case
Solr vs. Elasticsearch - Case by CaseAlexandre Rafalovitch The document compares Solr and Elasticsearch, highlighting their similarities and differences, including aspects like full-text search capabilities, configuration methods, and plugin usage. It discusses specific technical details about their setups, indexing processes, and search functionalities, illustrating the strengths and weaknesses of both systems. Additionally, it covers topics such as nested documents, cloud deployment strategies, and explicit versus dynamic mapping in each tool.
Elasticsearch Introduction
Elasticsearch IntroductionRoopendra Vishwakarma Elasticsearch is a distributed, open source search and analytics engine built on Apache Lucene. It allows storing and searching of documents of any schema in JSON format. Documents are organized into indexes which can have multiple shards and replicas for scalability and high availability. Elasticsearch provides a RESTful API and can be easily extended with plugins. It is widely used for full-text search, structured search, analytics and more in applications requiring real-time search and analytics of large volumes of data.
Elastic search apache_solr
Elastic search apache_solrmacrochen Elasticsearch offers several advantages over Apache Solr including being more easily distributed, replicated, and supporting real-time indexing. It allows for easy sharding and replication of indexes across multiple nodes. However, Elasticsearch lacks some features found in Solr such as spell checking, date math, and facet pagination. The document provides an overview of the similarities and differences between Elasticsearch and Solr for choosing between the two search servers.
An Introduction to Elastic Search.
An Introduction to Elastic Search.Jurriaan Persyn The document discusses technical details of implementing full-text search using PHP/MySQL and highlights advanced features of Elasticsearch, a distributed search engine built on Apache Lucene. It covers aspects such as data indexing, tokenization, stemming, filtering, and relevance scoring. Additionally, it compares Elasticsearch with other search engines like Solr and Sphinx, and outlines the benefits of using Elasticsearch for efficient and scalable search solutions.
Elasticsearch as a search alternative to a relational database
Elasticsearch as a search alternative to a relational databaseKristijan Duvnjak The document provides an introduction to Elasticsearch, highlighting its features as a distributed, schema-free search engine built on Apache Lucene, suitable for users transitioning from relational databases. It details the structure of Elasticsearch clusters, indexing, shards, and the importance of JSON-based documents, as well as describing how data retrieval and search functionalities work. Additionally, it outlines practical use cases and performance benefits of Elasticsearch in comparison to traditional databases.
Elasticsearch for Data Analytics
Elasticsearch for Data AnalyticsFelipe This document provides an overview of using Elasticsearch for data analytics. It discusses various aggregation techniques in Elasticsearch like terms, min/max/avg/sum, cardinality, histogram, date_histogram, and nested aggregations. It also covers mappings, dynamic templates, and general tips for working with aggregations. The main takeaways are that aggregations in Elasticsearch provide insights into data distributions and relationships similarly to GROUP BY in SQL, and that mappings and templates can optimize how data is indexed for aggregation purposes.
Introduction to Elasticsearch
Introduction to ElasticsearchSperasoft The document provides an overview of Elasticsearch including that it is easy to install, horizontally scalable, and highly available. It discusses Elasticsearch's core search capabilities using Lucene and how data can be stored and retrieved. The document also covers Elasticsearch's distributed nature, plugins, scripts, custom analyzers, and other features like aggregations, filtering and sorting.
Introduction to Elasticsearch
Introduction to ElasticsearchJason Austin Elasticsearch is a distributed, RESTful search and analytics engine that allows for fast searching, filtering, and analysis of large volumes of data. It is document-based and stores structured and unstructured data in JSON documents within configurable indices. Documents can be queried using a simple query string syntax or more complex queries using the domain-specific query language. Elasticsearch also supports analytics through aggregations that can perform metrics and bucketing operations on document fields.
Workshop: Learning Elasticsearch
Workshop: Learning ElasticsearchAnurag Patel Elasticsearch is a distributed, open source search and analytics engine. It allows storing and searching of documents of any schema in real-time. Documents are organized into indices which can contain multiple types of documents. Indices are partitioned into shards and replicas to allow horizontal scaling and high availability. The document consists of a JSON object which is indexed and can be queried using a RESTful API.
ElasticSearch: Distributed Multitenant NoSQL Datastore and Search Engine
ElasticSearch: Distributed Multitenant NoSQL Datastore and Search EngineDaniel N The document provides an overview of a distributed multitenant NoSQL datastore and search engine, emphasizing concepts such as SQL vs NoSQL, the architecture of Elasticsearch, and core components like indices, shards, and nodes. It highlights the capabilities of Elasticsearch, including cloud compatibility, RESTful API support, and its scalability options. Additionally, it discusses practical usage examples, typical queries, and integrations with AWS services.
Intro to Elasticsearch
Intro to ElasticsearchClifford James Elasticsearch is an open-source, distributed, real-time document indexer with support for online analytics. It has features like a powerful REST API, schema-less data model, full distribution and high availability, and advanced search capabilities. Documents are indexed into indexes which contain mappings and types. Queries retrieve matching documents from indexes. Analysis converts text into searchable terms using tokenizers, filters, and analyzers. Documents are distributed across shards and replicas for scalability and fault tolerance. The REST APIs can be used to index, search, and inspect the cluster.
Elasticsearch: You know, for search! and more!
Elasticsearch: You know, for search! and more!Philips Kokoh Prasetyo Elasticsearch is presented as an expert in real-time search, aggregation, and analytics. The document outlines Elasticsearch concepts like indexing, mapping, analysis, and the query DSL. Examples are provided for real-time search queries, aggregations including terms, date histograms, and geo distance. Lessons learned from using Elasticsearch at LARC are also discussed.
Introduction to Elasticsearch
Introduction to ElasticsearchRuslan Zavacky Elasticsearch is a distributed, schema-free document store designed for real-time data indexing and analytics, built on Apache Lucene. It provides full-text search capabilities, supports JSON documents, and features a RESTful API for operations, ensuring data safety and quick access. The platform is open-source under the Apache 2.0 license, making it highly extensible and accessible for various applications.
Elastic search
Elastic searchNexThoughts Technologies Elasticsearch is an open-source search engine and analytics engine built on Apache Lucene that allows for real-time distributed search across indexes and analytics capabilities. It consists of clusters of nodes that store indexed data and can search across the clusters. The data is divided into shards and replicas can be made of shards for redundancy. Elasticsearch supports different analyzers for tokenizing text and filtering searches.
Viewers also liked (6)
Presentation
Presentationkiarash1361 The document proposes a design for a durable, low-latency key-value data store. It discusses approaches to achieving durability like periodic snapshots and log replication. The proposed solution uses asynchronous log replication to remote storage units for durability and availability. The design uses chain replication of logs for synchronous writes and allows forming stable storage units across multiple machines for scalability. Evaluation shows the design can achieve high throughput and low latency even with additional replication.
Présentation de la solution Strategeex
Présentation de la solution StrategeexVisiativ Le document présente une solution logicielle, Strategeex, dédiée à l'analyse stratégique des organisations, permettant de formaliser des scénarios de développement et de piloter leur mise en œuvre. Il décrit également le processus d'analyse interne et externe, ainsi que la détermination des domaines d'activités stratégiques et l'évaluation de la position concurrentielle. Enfin, Strategeex facilite la formulation du diagnostic stratégique et la construction de plans d'amélioration à travers des outils de reporting et de pilotage adaptés.
ElasticSearch in action
ElasticSearch in actionCodemotion Elasticsearch is a full-text search engine and NoSQL analytics database built on Java and Lucene that supports real-time data processing and schema-less indexing. It offers RESTful APIs and features like inverted indices for enhanced search performance, and supports a variety of query types and mapping options for user-defined schemas. The document also covers specific operations, such as data insertion, mapping, and examples of various search and aggregation queries.
Data Exploration with Elasticsearch
Data Exploration with ElasticsearchAleksander Stensby The document provides an overview of data exploration using Elasticsearch, detailing its fundamentals, architecture, and operations such as indexing, querying, and filtering. It outlines how to implement searches and aggregations within the Elasticsearch environment while providing practical demos for installation and usage. Additional topics covered include relevancy ranking, tokenization, and advanced querying techniques.
Analyse des Sentiments -cas twitter- "Opinion Detection with Machine Lerning "
Analyse des Sentiments -cas twitter- "Opinion Detection with Machine Lerning "Soumia Elyakote HERMA Le document traite de l'analyse des sentiments sur Twitter, mettant en avant son utilité pour prédire le succès de produits et événements. Il présente différentes méthodes, notamment l'algorithme Naïve Bayes, pour classifier les sentiments à partir de données tweetées. Les résultats montrent que le modèle tf-idf surpasse le modèle booléen en performance, soulignant l'importance croissante de cette technique dans la détection d'opinions.
Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...
Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...StampedeCon This document compares various HDFS data storage formats, including Avro, Parquet, and ORC, focusing on their use cases, strengths, and weaknesses. It outlines considerations for choosing a format based on write and read requirements, schema evolution, and data types. The document concludes with examples and practical implications of schema evolution in data management.
Ad
Similar to Scaling Analytics with elasticsearch (20)
PEARC17: Designsafe: Using Elasticsearch to Share and Search Data on a Scienc...
PEARC17: Designsafe: Using Elasticsearch to Share and Search Data on a Scienc...Josue Balandrano The document discusses the DesignSafe platform, focused on natural hazards engineering research, and how it utilizes Elasticsearch for data sharing and searching. It explains the architecture of DesignSafe, including Django middleware, Agave, and various data depot functionalities that aid users in managing and querying data effectively. Key features of Elasticsearch are highlighted, including its capabilities for search efficiency and flexibility in handling various types of queries.
ELK - What's new and showcases
ELK - What's new and showcasesAndrii Gakhov The document discusses recent updates and features of Elasticsearch and Kibana, highlighting the open-source search engine's scalable design, enhanced security with Shield, and new visualization capabilities. Key features include user authentication, advanced aggregation methods, a new user interface, and SSL encryption for secure communications. It also presents various aggregation examples and solutions for analyzing data, focusing on performance improvements and new analytic capabilities since version 1.2.0.
ElasticSearch as (only) datastore
ElasticSearch as (only) datastoreTomas Sirny The document provides a comprehensive guide on integrating Elasticsearch as a datastore for projects, particularly focusing on its application in a movie recommendation service. It outlines the pros and cons of using Elasticsearch, including its powerful search capabilities and the need for reindexing. Best practices for setting up clusters, indexing, and data management using different backends are discussed to ensure efficient operation.
Elastic Search
Elastic SearchLukas Vlcek This document provides an overview of ElasticSearch, an open source, distributed, RESTful search and analytics engine. It discusses how ElasticSearch is highly available, distributed across shards and replicas, and can be deployed in the cloud. Examples are provided showing how to index and search data via the REST API and retrieve cluster health information. Advanced features like faceting, scripting, parent/child relationships, and versioning are also summarized.
Elasticsearch speed is key
Elasticsearch speed is keyEnterprise Search Warsaw Meetup The document provides a comprehensive overview of Elasticsearch, describing it as an open-source, distributed, scalable, and highly available search engine with real-time analytics capabilities. It discusses key aspects such as cluster management, the importance of speed in querying and indexing, and various features like shards, filtering, and aggregations. Additionally, it highlights operational considerations and best practices for optimizing performance and managing resources effectively.
The Power of Elasticsearch
The Power of ElasticsearchInfochimps, a CSC Big Data Business Elasticsearch is a scalable and fault-tolerant database built on Apache Lucene, designed to handle large volumes of data while providing powerful full-text search capabilities. It offers advanced features such as geo queries, complex document structures, and an easy-to-use REST API, making it suitable for building various applications. Its architecture allows for robust administration and monitoring, ensuring high availability and performance for complex data operations.
Introducing ElasticSearch - Ashish
Introducing ElasticSearch - AshishEntrepreneur / Startup The document provides an introduction to Elasticsearch, a distributed real-time search and analytics engine known for its scalability and high availability. It explains the RESTful API, CRUD operations for managing data, as well as various search functionalities and data mappings within Elasticsearch. Additional details include the use of analyzers for text processing and how to optimize search queries.
ElasticSearch
ElasticSearchVolodymyr Kraietskyi Elasticsearch is a search engine based on Apache Lucene that provides distributed, full-text search capabilities. It allows users to store and search documents of any structure in near real-time. Documents are organized into indexes, shards, and clusters to provide scalability and fault tolerance. Elasticsearch uses analysis and mapping to index documents for full-text search. Queries can be built using the Elasticsearch DSL for complex searches. While Elasticsearch provides fast search, it has disadvantages for transactional operations or large document churn. Elastic HQ is a web plugin that provides monitoring and management of Elasticsearch clusters through a browser-based interface.
Elasticsearch
ElasticsearchYervand Aghababyan The document provides an overview of Elasticsearch, comparing it to traditional SQL databases and discussing its advantages in handling complex, flexible searches and indexing. Key features include full-text indexing, relevance scoring, geocoding, and the ability to manage nested documents, with an emphasis on ease of use and scalability. It also addresses common challenges in search functionality and the importance of performance monitoring and optimization.
Semantic Search overview at SSSW 2012
Semantic Search overview at SSSW 2012Peter Mika This document discusses semantic search and how it can improve traditional information retrieval systems. It provides examples of how semantic search uses structured data and schemas to better understand user intent and content meaning. This allows semantic search to enhance various stages of the information retrieval process from query interpretation to result presentation. The document also outlines the growing adoption of semantic web standards like RDFa and schema.org to expose structured data on webpages.
Philly PHP: April '17 Elastic Search Introduction by Aditya Bhamidpati
Philly PHP: April '17 Elastic Search Introduction by Aditya BhamidpatiRobert Calcavecchia The document provides an introduction to Elasticsearch, detailing its functionality as an open-source search engine built on Apache Lucene, which enables full-text search and real-time analytics. Key features include scalability, a distributed architecture, and communication via HTTP REST API, along with its capability to handle schema-less JSON documents. Furthermore, it discusses various search capabilities, types of queries, and some limitations of Elasticsearch, emphasizing its use by several prominent companies.
Large-Scale Semantic Search
Large-Scale Semantic SearchRoi Blanco The document discusses large-scale semantic search and its importance in improving user queries and search results. It highlights the challenges in understanding user queries, the complexity of unstructured data, and the need for efficient indexing methods to support semantic analysis. Additionally, it covers various applications of semantic search, including enhanced search capabilities, recommender systems, and the integration of structured and unstructured data.
Semantic Search Tutorial at SemTech 2012
Semantic Search Tutorial at SemTech 2012 Thanh Tran This document provides an overview of a seminar on semantic search. It introduces the speakers, Peter Mika and Tran Duc Thanh, and outlines the agenda which includes introductions to semantic web data using the RDF data model, crawling and indexing RDF data, query processing, ranking results, and evaluating semantic search. It discusses why semantic search is needed to address queries that are not well solved by traditional search and provides examples. It also describes combining document retrieval with data retrieval from structured sources and how semantic search systems can incorporate different models and techniques.
ElasticSearch AJUG 2013
ElasticSearch AJUG 2013Roy Russo This document provides an introduction to Elasticsearch, covering the basics, concepts, data structure, inverted index, REST API, bulk API, percolator, Java integration, and topics not covered. It discusses how Elasticsearch is a document-oriented search engine that allows indexing and searching of JSON documents without schemas. Documents are distributed across shards and replicas for horizontal scaling and high availability. The REST API and query DSL allow full-text search and filtering of documents.
ElasticSearch on AWS - Real Estate portal case study (Spitogatos.gr)
ElasticSearch on AWS - Real Estate portal case study (Spitogatos.gr) Andreas Chatzakis The document presents a case study on the implementation of Elasticsearch for the real estate portal Spitogatos.gr, aimed at improving property search in Greece. It outlines the challenges of property searches, the decision-making process for selecting Elasticsearch over other solutions, and describes the architecture, indexing, and operational aspects of the deployment on AWS. Key features include the ability to provide complex and flexible search capabilities, faceted navigation, and scalable infrastructure.
Elastic Search Capability Presentation.pptx
Elastic Search Capability Presentation.pptxKnoldus Inc. The document outlines the importance of etiquette during sessions, emphasizing punctuality, feedback, and minimizing disturbances. It provides an overview of Elasticsearch and the Elastic Stack components, including their uses in data analytics, searching, and monitoring. Additionally, it covers data ingestion methods, query DSL for constructing searches, and performance strategies for managing large datasets.
quick intro to elastic search
quick intro to elastic search medcl ElasticSearch is an open source, distributed, RESTful search and analytics engine. It allows storage and search of documents in near real-time. Documents are indexed and stored across multiple nodes in a cluster. The documents can be queried using a RESTful API or client libraries. ElasticSearch is built on top of Lucene and provides scalability, reliability and availability.
ElasticSearch Basic Introduction
ElasticSearch Basic IntroductionMayur Rathod Elasticsearch is an open-source, distributed search and analytics engine built on Apache Lucene, featuring a RESTful API for easy scalability and complex querying. It supports multi-tenancy and document-oriented storage with a schema-free design, enabling advanced search capabilities, including full-text search and aggregations. The software has an active community, allowing for extensive configurability and extensibility while ensuring data safety through optimistic version control.
Elasticsearch in 15 Minutes
Elasticsearch in 15 MinutesKarel Minarik The document provides an overview of using Elasticsearch. It demonstrates how to install Elasticsearch, index and update documents, perform searches, add nodes to the cluster, and configure shards and clusters. It also covers JSON and HTTP usage, different search types (terms, phrases, wildcards), filtering and boosting searches, and the JSON query DSL. Faceted searching is demonstrated using terms and terms_stats facets. Advanced topics like mapping, analyzing, and features above the basic search capabilities are also briefly mentioned.
Bridging Batch and Real-time Systems for Anomaly Detection
Bridging Batch and Real-time Systems for Anomaly DetectionDataWorks Summit This document discusses using a stack of Hadoop, Spark, and Elasticsearch to perform anomaly detection on large datasets in both batch and real-time. Hadoop is used for large-scale data storage and preprocessing. Spark is used to perform in-depth analysis to identify common entities and build models. Elasticsearch allows searching the data in real-time and performing aggregations to identify uncommon entities. A live loop continuously adapts the models to react to streaming data and improve anomaly detection over time.
Ad
Recently uploaded (20)
Smarter Aviation Data Management: Lessons from Swedavia Airports and Sweco
Smarter Aviation Data Management: Lessons from Swedavia Airports and SwecoSafe Software Managing airport and airspace data is no small task, especially when you’re expected to deliver it in AIXM format without spending a fortune on specialized tools. But what if there was a smarter, more affordable way?
Join us for a behind-the-scenes look at how Sweco partnered with Swedavia, the Swedish airport operator, to solve this challenge using FME and Esri.
Learn how they built automated workflows to manage periodic updates, merge airspace data, and support data extracts – all while meeting strict government reporting requirements to the Civil Aviation Administration of Sweden.
Even better? Swedavia built custom services and applications that use the FME Flow REST API to trigger jobs and retrieve results – streamlining tasks like securing the quality of new surveyor data, creating permdelta and baseline representations in the AIS schema, and generating AIXM extracts from their AIS data.
To conclude, FME expert Dean Hintz will walk through a GeoBorders reading workflow and highlight recent enhancements to FME’s AIXM (Aeronautical Information Exchange Model) processing and interpretation capabilities.
Discover how airports like Swedavia are harnessing the power of FME to simplify aviation data management, and how you can too.
"Database isolation: how we deal with hundreds of direct connections to the d...
"Database isolation: how we deal with hundreds of direct connections to the d...Fwdays What can go wrong if you allow each service to access the database directly? In a startup, this seems like a quick and easy solution, but as the system scales, problems appear that no one could have guessed.
In my talk, I'll share Solidgate's experience in transforming its architecture: from the chaos of direct connections to a service-based data access model. I will talk about the transition stages, bottlenecks, and how isolation affected infrastructure support. I will honestly show what worked and what didn't. In short, we will analyze the controversy of this talk.
9-1-1 Addressing: End-to-End Automation Using FME
9-1-1 Addressing: End-to-End Automation Using FMESafe Software This session will cover a common use case for local and state/provincial governments who create and/or maintain their 9-1-1 addressing data, particularly address points and road centerlines. In this session, you'll learn how FME has helped Shelby County 9-1-1 (TN) automate the 9-1-1 addressing process; including automatically assigning attributes from disparate sources, on-the-fly QAQC of said data, and reporting. The FME logic that this presentation will cover includes: Table joins using attributes and geometry, Looping in custom transformers, Working with lists and Change detection.
OWASP Barcelona 2025 Threat Model Library
OWASP Barcelona 2025 Threat Model LibraryPetraVukmirovic Threat Model Library Launch at OWASP Barcelona 2025
https://owasp.org/www-project-threat-model-library/
GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now.pdf
GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now.pdfPriyanka Aash GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now
The Future of Product Management in AI ERA.pdf
The Future of Product Management in AI ERA.pdfAlyona Owens Hi, I’m Aly Owens, I have a special pleasure to stand here as over a decade ago I graduated from CityU as an international student with an MBA program. I enjoyed the diversity of the school, ability to work and study, the network that came with being here, and of course the price tag for students here has always been more affordable than most around.
Since then I have worked for major corporations like T-Mobile and Microsoft and many more, and I have founded a startup. I've also been teaching product management to ensure my students save time and money to get to the same level as me faster avoiding popular mistakes. Today as I’ve transitioned to teaching and focusing on the startup, I hear everybody being concerned about Ai stealing their jobs… We’ll talk about it shortly.
But before that, I want to take you back to 1997. One of my favorite movies is “Fifth Element”. It wowed me with futuristic predictions when I was a kid and I’m impressed by the number of these predictions that have already come true. Self-driving cars, video calls and smart TV, personalized ads and identity scanning. Sci-fi movies and books gave us many ideas and some are being implemented as we speak. But we often get ahead of ourselves:
Flying cars,Colonized planets, Human-like AI: not yet, Time travel, Mind-machine neural interfaces for everyone: Only in experimental stages (e.g. Neuralink).
Cyberpunk dystopias: Some vibes (neon signs + inequality + surveillance), but not total dystopia (thankfully).
On the bright side, we predict that the working hours should drop as Ai becomes our helper and there shouldn’t be a need to work 8 hours/day. Nobody knows for sure but we can require that from legislation. Instead of waiting to see what the government and billionaires come up with, I say we should design our own future.
So, we as humans, when we don’t know something - fear takes over. The same thing happened during the industrial revolution. In the Industrial Era, machines didn’t steal jobs—they transformed them but people were scared about their jobs. The AI era is making similar changes except it feels like robots will take the center stage instead of a human. First off, even when it comes to the hottest space in the military - drones, Ai does a fraction of work. AI algorithms enable real-time decision-making, obstacle avoidance, and mission optimization making drones far more autonomous and capable than traditional remote-controlled aircraft. Key technologies include computer vision for object detection, GPS-enhanced navigation, and neural networks for learning and adaptation. But guess what? There are only 2 companies right now that utilize Ai in drones to make autonomous decisions - Skydio and DJI.
Curietech AI in action - Accelerate MuleSoft development
Curietech AI in action - Accelerate MuleSoft developmentshyamraj55 CurieTech AI in Action – Accelerate MuleSoft Development
Overview:
This presentation demonstrates how CurieTech AI’s purpose-built agents empower MuleSoft developers to create integration workflows faster, more accurately, and with less manual effort
linkedin.com
+12
curietech.ai
+12
meetups.mulesoft.com
+12
.
Key Highlights:
Dedicated AI agents for every stage: Coding, Testing (MUnit), Documentation, Code Review, and Migration
curietech.ai
+7
curietech.ai
+7
medium.com
+7
DataWeave automation: Generate mappings from tables or samples—95%+ complete within minutes
linkedin.com
+7
curietech.ai
+7
medium.com
+7
Integration flow generation: Auto-create Mule flows based on specifications—speeds up boilerplate development
curietech.ai
+1
medium.com
+1
Efficient code reviews: Gain intelligent feedback on flows, patterns, and error handling
youtube.com
+8
curietech.ai
+8
curietech.ai
+8
Test & documentation automation: Auto-generate MUnit test cases, sample data, and detailed docs from code
curietech.ai
+5
curietech.ai
+5
medium.com
+5
Why Now?
Achieve 10× productivity gains, slashing development time from hours to minutes
curietech.ai
+3
curietech.ai
+3
medium.com
+3
Maintain high accuracy with code quality matching or exceeding manual efforts
curietech.ai
+2
curietech.ai
+2
curietech.ai
+2
Ideal for developers, architects, and teams wanting to scale MuleSoft projects with AI efficiency
Conclusion:
CurieTech AI transforms MuleSoft development into an AI-accelerated workflow—letting you focus on innovation, not repetition.
Wenn alles versagt - IBM Tape schützt, was zählt! Und besonders mit dem neust...
Wenn alles versagt - IBM Tape schützt, was zählt! Und besonders mit dem neust...Josef Weingand IBM LTO10
OpenACC and Open Hackathons Monthly Highlights June 2025
OpenACC and Open Hackathons Monthly Highlights June 2025OpenACC The OpenACC organization focuses on enhancing parallel computing skills and advancing interoperability in scientific applications through hackathons and training. The upcoming 2025 Open Accelerated Computing Summit (OACS) aims to explore the convergence of AI and HPC in scientific computing and foster knowledge sharing. This year's OACS welcomes talk submissions from a variety of topics, from Using Standard Language Parallelism to Computer Vision Applications. The document also highlights several open hackathons, a call to apply for NVIDIA Academic Grant Program and resources for optimizing scientific applications using OpenACC directives.
Techniques for Automatic Device Identification and Network Assignment.pdf
Techniques for Automatic Device Identification and Network Assignment.pdfPriyanka Aash Techniques for Automatic Device Identification and Network Assignment
Using the SQLExecutor for Data Quality Management: aka One man's love for the...
Using the SQLExecutor for Data Quality Management: aka One man's love for the...Safe Software The SQLExecutor is one of FME’s most powerful and flexible transformers. Pivvot maintains a robust internal metadata hierarchy used to support ingestion and curation of thousands of external data sources that must be managed for quality before entering our platform. By using the SQLExecutor, Pivvot can efficiently detect problems and perform analysis before data is extracted from our staging environment, removing the need for rollbacks or cycles waisted on a failed job. This presentation will walk through three distinct examples of how Pivvot uses the SQLExecutor to engage its metadata hierarchy and integrate with its Data Quality Management workflows efficiently and within the source postgres database. Spatial Validation –Validating spatial prerequisites before entering a production environment. Reference Data Validation - Dynamically validate domain-ed columns across any table and multiple columns per table. Practical De-duplication - Removing identical or near-identical well point locations from two distinct source datasets in the same table.
Securing Account Lifecycles in the Age of Deepfakes.pptx
Securing Account Lifecycles in the Age of Deepfakes.pptxFIDO Alliance Securing Account Lifecycles in the Age of Deepfakes
cnc-processing-centers-centateq-p-110-en.pdf
cnc-processing-centers-centateq-p-110-en.pdfAmirStern2 מרכז עיבודים תעשייתי בעל 3/4/5 צירים, עד 22 החלפות כלים עם כל אפשרויות העיבוד הדרושות. בעל שטח עבודה גדול ומחשב נוח וקל להפעלה בשפה העברית/רוסית/אנגלית/ספרדית/ערבית ועוד..
מסוגל לבצע פעולות עיבוד שונות המתאימות לענפים שונים: קידוח אנכי, אופקי, ניסור, וכרסום אנכי.
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...Edge AI and Vision Alliance For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2025/06/mpu-a-transformative-solution-for-next-gen-ai-at-the-edge-a-presentation-from-fotonation/
Petronel Bigioi, CEO of FotoNation, presents the “MPU+: A Transformative Solution for Next-Gen AI at the Edge” tutorial at the May 2025 Embedded Vision Summit.
In this talk, Bigioi introduces MPU+, a novel programmable, customizable low-power platform for real-time, localized intelligence at the edge. The platform includes an AI-augmented image signal processor that enables leading image and video quality.
In addition, it integrates ultra-low-power object and motion detection capabilities to enable always-on computer vision. A programmable neural processor provides flexibility to efficiently implement new neural networks. And additional specialized engines facilitate image stabilization and audio enhancements.
AI vs Human Writing: Can You Tell the Difference?
AI vs Human Writing: Can You Tell the Difference?Shashi Sathyanarayana, Ph.D This slide illustrates a side-by-side comparison between human-written, AI-written, and ambiguous content. It highlights subtle cues that help readers assess authenticity, raising essential questions about the future of communication, trust, and thought leadership in the age of generative AI.
CapCut Pro Crack For PC Latest Version {Fully Unlocked} 2025
CapCut Pro Crack For PC Latest Version {Fully Unlocked} 2025pcprocore 👉𝗡𝗼𝘁𝗲:𝗖𝗼𝗽𝘆 𝗹𝗶𝗻𝗸 & 𝗽𝗮𝘀𝘁𝗲 𝗶𝗻𝘁𝗼 𝗚𝗼𝗼𝗴𝗹𝗲 𝗻𝗲𝘄 𝘁𝗮𝗯> https://pcprocore.com/ 👈◀
CapCut Pro Crack is a powerful tool that has taken the digital world by storm, offering users a fully unlocked experience that unleashes their creativity. With its user-friendly interface and advanced features, it’s no wonder why aspiring videographers are turning to this software for their projects.
OpenPOWER Foundation & Open-Source Core Innovations
OpenPOWER Foundation & Open-Source Core InnovationsIBM penPOWER offers a fully open, royalty-free CPU architecture for custom chip design.
It enables both lightweight FPGA cores (like Microwatt) and high-performance processors (like POWER10).
Developers have full access to source code, specs, and tools for end-to-end chip creation.
It supports AI, HPC, cloud, and embedded workloads with proven performance.
Backed by a global community, it fosters innovation, education, and collaboration.
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...Edge AI and Vision Alliance
Scaling Analytics with elasticsearch
- 1. Scaling Analytics with elasticsearch Dan Noble @dwnoble
- 2. Background • Technologist at The HumanGeo • We use elasticsearch to build social media analysis tools • 100MM documents indexed • 600GB+ index size • Author of Python elasticsearch driver “rawes” https://github.com/humangeo/rawes
- 3. Overview • What is elasticsearch? • Scaling with elasticsearch • How can I use elasticsearch to help with analytics? • Use Case: Social Media Analytics
- 5. Search Engine • Open source • Distributed • Automatic failover • Crazy fast
- 6. Search Engine • Actively maintained • REST API • JSON messages • Lucene based
- 7. Search Elasticsearch “Cluster” Host Index: Articles • Simple case: one host • One index containing a set of articles
- 8. Distributed Search Elasticsearch “Cluster” Host Host Articles (a) Articles (b) • Too much data? • Add another host • Indices can be broken up into “shards” and live on different machines
- 9. Redundancy Elasticsearch Cluster Host Host Articles (a) Articles (b) Articles (b) Articles (a) • Shards can be replicated to improve availability
- 10. Node Auto Discovery Elasticsearch Cluster Host Host Host Articles (a) Articles (b) Articles (b) Articles (b) Articles (a) Articles (a) • Say we add a third host • elasticsearch will automatically start moving shards to this new host to distribute load
- 11. Failover Elasticsearch Cluster Host Host Host Articles (a) Articles (b) Articles (b) Articles (b) Articles (a) Articles (a) • Say a host goes down • Shards on that host are no longer available for search • Elasticsearch automatically rebuilds these two shards on other hosts
- 12. Querying Elasticsearch Cluster Host Host Host Articles (a) Articles (b) Articles (b) Articles(a) Query: “Barack Obama” Can query against Client Search for articles any host (Web Application) Send request to other shards if needed
- 13. REST API • JSON query syntax • Developer friendly • Easy to get started
- 14. Python Example import rawes es = rawes.Elastic('elastic-00:9200') es.get('articles/_search', data={ "query": { "filtered" : { "query" : { "query_string" : { "query" : "Barack Obama" } } } } })
- 15. Community
- 16. Elasticsearch Summary • Scales horizontally • Redundancy • Configures itself automatically • Developer friendly
- 17. Analytics and elasticsearch • Date Histograms • Statistical facets • Geospatial queries • All with arbitrary search parameters • Again: Fast
- 18. Use Case: Social Media Analysis • Use social media APIs to search for data on a topic of interest • 100MM documents indexed • Sentiment analysis • Location extraction (“Geotagging”)
- 19. Sample Document es.post('articles/facebook', data={ ”date": "2012-09-01 08:37:55", "tags": { "sentiment": { "positive": 0.36, "negative": 0.10 } "geotags": [{ "term" : "Cairo", "location" : "30.0566,31.2262”, “type” : “geo_point” }], "search_terms": [ "Mohamed Morsi" ] }, "item": { "publisher: "Facebook" "source_domain": "www.facebook.com", "author": "James Smith", "source_url": "http://www.facebook.com/5551231234/posts/414141414141", "content_text": "Mohamed Morsi visits Iran for first time since 1979 ....", "title": "James Smith posted a note to Facebook", "author_url: "http://www.facebook.com/profile.php?id=5551231234" } })
- 21. Date Histogram for Sentiment es.get('articles/_search', data= { "query" : { "query_string" : { "query" : "Mohamed Morsi" } }, "facets" : { "sentiment_histogram" : { "date_histogram" : { "key_field" : "date_of_information.$date", "value_field" : "tags.sentiment.positive", "interval" : "day" } } } })
- 22. Date Histogram for Sentiment
- 23. Statistical Facet for Sentiment: Query es.get('articles/_search', data={ "query" : { "query_string" : { "query" : "Mohamed Morsi" } }, "facets" : { "sentiment_stats" : { "statistical" : { "field" : "tags.sentiment.positive" } } } })
- 24. Statistical Facet for Sentiment: Result { "facets": { "sentiment_stats": { "_type": "statistical", "count": 8825, "max": 0.375, "mean": 0.008503991588291782, "min": 0.0, "std_deviation": 0.021251077265305472, "sum_of_squares": 4.623648343200283, "total": 75.04772576667497, "variance": 0.00045160828493598306 } }, "hits": { "hits": [], "max_score": 1.1120162, "total": 8825 }, "took": 60 }
- 25. Top Keywords es.get('articles/_search', data={ "query" : { "match_all" : {} }, "facets" : { "search_terms" : { "terms" : { "field" : "tags.search_terms", "size" : 3 } } } })
- 26. Top Search Terms
- 27. Geospatial search es.get('articles/_search', data={ "query" : { "filtered" : { "filter" : { "geo_distance" : { "distance" : ”20km", "tags.geotags.location" : { "lat" : 30, "lon" : 31 } } } } } })
- 28. Questions