Elasticsearch::Persistence
¶ ↑
Persistence layer for Ruby domain objects in Elasticsearch
, using the Repository pattern.
Compatibility¶ ↑
This library is compatible with Ruby 3.1 and higher.
The version numbers follow the Elasticsearch
major versions. Currently the main
branch is compatible with version 8.x
of the Elasticsearch
stack.
Rubygem | Elasticsearch |
|
---|---|---|
0.1 | → | 1.x |
2.x | → | 2.x |
5.x | → | 5.x |
6.x | → | 6.x |
7.x | → | 7.x |
8.x | → | 8.x |
main | → | 8.x |
Installation¶ ↑
Install the package from Rubygems:
gem install elasticsearch-persistence
To use an unreleased version, either add it to your Gemfile
for Bundler:
gem 'elasticsearch-persistence', git: 'git://github.com/elastic/elasticsearch-rails.git', branch: '6.x'
or install it from a source code checkout:
git clone https://github.com/elastic/elasticsearch-rails.git cd elasticsearch-rails/elasticsearch-persistence bundle install rake install
Usage¶ ↑
The library provides the Repository pattern for adding persistence to your Ruby objects.
The Repository Pattern¶ ↑
The Elasticsearch::Persistence::Repository
module provides an implementation of the repository pattern and allows you to save, delete, find and search objects stored in Elasticsearch
, as well as configure mappings and settings for the index. It’s an unobtrusive and decoupled way of adding persistence to your Ruby objects.
Let’s have a simple plain old Ruby object (PORO):
class Note attr_reader :attributes def initialize(attributes={}) @attributes = attributes end def to_hash @attributes end end
Let’s create a default, “dumb” repository, as a first step:
require 'elasticsearch/persistence' class MyRepository; include Elasticsearch::Persistence::Repository; end repository = MyRepository.new
We can save a Note
instance into the repository…
note = Note.new id: 1, text: 'Test' repository.save(note) # PUT http://localhost:9200/repository/_doc/1 [status:201, request:0.210s, query:n/a] # > {"id":1,"text":"Test"} # < {"_index":"repository","_id":"1","_version":1,"created":true}
…find it…
n = repository.find(1) # GET http://localhost:9200/repository/_doc/1 [status:200, request:0.003s, query:n/a] # < {"_index":"repository","_id":"1","_version":2,"found":true, "_source" : {"id":1,"text":"Test"}} => <Note0x007fcbfc0c4980 @attributes={"id"=>1, "text"=>"Test"}>
…search for it…
repository.search(query: { match: { text: 'test' } }).first # GET http://localhost:9200/repository/_search [status:200, request:0.005s, query:0.002s] # > {"query":{"match":{"text":"test"}}} # < {"took":2, ... "hits":{"total":1, ... "hits":[{ ... "_source" : {"id":1,"text":"Test"}}]}} => <Note0x007fcbfc1c7b70 @attributes={"id"=>1, "text"=>"Test"}>
…or delete it:
repository.delete(note) # DELETE http://localhost:9200/repository/_doc/1 [status:200, request:0.014s, query:n/a] # < {"found":true,"_index":"repository","_id":"1","_version":3} => {"found"=>true, "_index"=>"repository", "_id"=>"1", "_version"=>2}
The repository module provides a number of features and facilities to configure and customize the behavior:
-
Configuring the
Elasticsearch
client being used -
Setting the index name, and object class for deserialization
-
Composing mappings and settings for the index
-
Creating, deleting or refreshing the index
-
Finding or searching for documents
-
Providing access both to domain objects and hits for search results
-
Providing access to the
Elasticsearch
response for search results (aggregations, total, …) -
Defining the methods for serialization and deserialization
There are two mixins you can include in your Repository class. The first Elasticsearch::Persistence::Repository
, provides the basic methods and settings you’ll need. The second, Elasticsearch::Persistence::Repository::DSL
adds some additional class methods that allow you to set options that instances of the class will share.
Basic Repository mixin¶ ↑
For simple cases, you can just include the Elasticsearch::Persistence::Repository
mixin to your class:
class MyRepository include Elasticsearch::Persistence::Repository # Customize the serialization logic def serialize(document) super.merge(my_special_key: 'my_special_stuff') end # Customize the de-serialization logic def deserialize(document) puts "# ***** CUSTOM DESERIALIZE LOGIC... *****" super end end client = Elasticsearch::Client.new(url: ENV['ELASTICSEARCH_URL'], log: true) repository = MyRepository.new(client: client, index_name: :my_notes, klass: Note) repository.settings number_of_shards: 1 do mapping do indexes :text, analyzer: 'snowball' end end
The custom Elasticsearch
client will be used now, with a custom index, as well as the custom serialization and de-serialization logic.
We can create the index with the desired settings and mappings:
repository.create_index! force: true # PUT http://localhost:9200/my_notes # > {"settings":{"number_of_shards":1},"mappings":{ ... {"text":{"analyzer":"snowball","type":"string"}}}}}
Save the document with extra properties added by the serialize
method:
repository.save(note) # PUT http://localhost:9200/my_notes/note/1 # > {"id":1,"text":"Test","my_special_key":"my_special_stuff"} {"_index"=>"my_notes", "_id"=>"1", "_version"=>4, ... }
And deserialize
it:
repository.find(1) # ***** CUSTOM DESERIALIZE LOGIC... ***** <Note0x007f9bd782b7a0 @attributes={... "my_special_key"=>"my_special_stuff"}>
The DSL mixin¶ ↑
In some cases, you’ll want to set some of the repository configurations at the class level. This makes most sense when the instances of the repository will use that same configuration:
require 'base64' class NoteRepository include Elasticsearch::Persistence::Repository include Elasticsearch::Persistence::Repository::DSL index_name 'notes' klass Note settings number_of_shards: 1 do mapping do indexes :text, analyzer: 'snowball' # Do not index images indexes :image, index: false end end # Base64 encode the "image" field in the document # def serialize(document) hash = document.to_hash.clone hash['image'] = Base64.encode64(hash['image']) if hash['image'] hash.to_hash end # Base64 decode the "image" field in the document # def deserialize(document) hash = document['_source'] hash['image'] = Base64.decode64(hash['image']) if hash['image'] klass.new hash end end
You can create an instance of this custom class and get each of the configurations.
client = Elasticsearch::Client.new(url: 'http://localhost:9200', log: true) repository = NoteRepository.new(client: client) repository.index_name # => 'notes'
You can also override the default configuration with options passed to the initialize method:
client = Elasticsearch::Client.new(url: 'http://localhost:9250', log: true) client.transport.logger.formatter = proc { |s, d, p, m| "\e[2m# #{m}\n\e[0m" } repository = NoteRepository.new(client: client, index_name: 'notes_development') repository.create_index!(force: true) note = Note.new('id' => 1, 'text' => 'Document with image', 'image' => '... BINARY DATA ...') repository.save(note) # PUT http://localhost:9200/notes_development/_doc/1 # > {"id":1,"text":"Document with image","image":"Li4uIEJJTkFSWSBEQVRBIC4uLg==\n"} puts repository.find(1).attributes['image'] # GET http://localhost:9200/notes_development/_doc/1 # < {... "_source" : { ... "image":"Li4uIEJJTkFSWSBEQVRBIC4uLg==\n"}} # => ... BINARY DATA ...
Functionality Provided by the Repository mixin¶ ↑
Each of the following configurations can be set for a repository instance. If you have included the Elasticsearch::Persistence::Repository::DSL
mixin, then you can use the class-level DSL methods to set each value. You can still override the configuration for any instance by passing options to the #initialize
method. Even if you don’t use the DSL mixin, you can set the instance configuration with options passed the #initialize
method.
Client¶ ↑
The repository uses the standard Elasticsearch
client.
client = Elasticsearch::Client.new(url: 'http://search.server.org') repository = NoteRepository.new(client: client) repository.client.transport.logger = Logger.new(STDERR) repository.client # => Elasticsearch::Client
or with the DSL mixin:
class NoteRepository include Elasticsearch::Persistence::Repository include Elasticsearch::Persistence::Repository::DSL client Elasticsearch::Client.new url: 'http://search.server.org' end repository = NoteRepository.new repository.client # => Elasticsearch::Client
Naming¶ ↑
The index_name
method specifies the Elasticsearch
index to use for storage, lookup and search. The default index name is ‘repository’.
repository = NoteRepository.new(index_name: 'notes_development') repository.index_name # => 'notes_development'
or with the DSL mixin:
class NoteRepository include Elasticsearch::Persistence::Repository include Elasticsearch::Persistence::Repository::DSL index_name 'notes_development' end repository = NoteRepository.new repository.index_name # => 'notes_development'
The klass
method specifies the Ruby class name to use when initializing objects from documents retrieved from the repository. If this value is not set, a Hash representation of the document will be returned instead.
repository = NoteRepository.new(klass: Note) repository.klass # => Note
or with the DSL mixin:
class NoteRepository include Elasticsearch::Persistence::Repository include Elasticsearch::Persistence::Repository::DSL klass Note end repository = NoteRepository.new repository.klass # => Note
Index Configuration¶ ↑
The settings
and mappings
methods, provided by the {elasticsearch-model
} gem, allow you to configure the index properties:
repository.settings number_of_shards: 1 repository.settings.to_hash # => {:number_of_shards=>1} repository.mappings { indexes :title, analyzer: 'snowball' } repository.mappings.to_hash # => { :note => {:properties=> ... }}
or with the DSL mixin:
class NoteRepository include Elasticsearch::Persistence::Repository include Elasticsearch::Persistence::Repository::DSL mappings { indexes :title, analyzer: 'snowball' } settings number_of_shards: 1 end repository = NoteRepository.new
Create a Repository and set its configuration with a block¶ ↑
You can also use the #create
method to instantiate and set the mappings and settings on an instance with a block in one call:
repository = NoteRepository.create(index_name: 'notes_development') do settings number_of_shards: 1, number_of_replicas: 0 do mapping dynamic: 'strict' do indexes :foo do indexes :bar end indexes :baz end end end
Index Management¶ ↑
The convenience methods create_index!
, delete_index!
and refresh_index!
allow you to manage the index lifecycle. These methods can only be called on repository instances and are not implemented at the class level.
Serialization¶ ↑
The serialize
and deserialize
methods allow you to customize the serialization of the document when it is persisted to Elasticsearch
, and define the initialization procedure when loading it from the storage:
class NoteRepository include Elasticsearch::Persistence::Repository def serialize(document) Hash[document.to_hash.map() { |k,v| v.upcase! if k == :title; [k,v] }] end def deserialize(document) MyNote.new ActiveSupport::HashWithIndifferentAccess.new(document['_source']).deep_symbolize_keys end end
Storage¶ ↑
The save
method allows you to store a domain object in the repository:
note = Note.new id: 1, title: 'Quick Brown Fox' repository.save(note) # => {"_index"=>"notes_development", "_id"=>"1", "_version"=>1, "created"=>true}
The update
method allows you to perform a partial update of a document in the repository. Use either a partial document:
repository.update id: 1, title: 'UPDATED', tags: [] # => {"_index"=>"notes_development", "_id"=>"1", "_version"=>2}
Or a script (optionally with parameters):
repository.update 1, script: 'if (!ctx._source.tags.contains(t)) { ctx._source.tags += t }', params: { t: 'foo' } # => {"_index"=>"notes_development", "_id"=>"1", "_version"=>3}
The delete
method allows you to remove objects from the repository (pass either the object itself or its ID):
repository.delete(note) repository.delete(1)
Finding¶ ↑
The find
method allows you to find one or many documents in the storage and returns them as deserialized Ruby objects:
repository.save Note.new(id: 2, title: 'Fast White Dog') note = repository.find(1) # => <MyNote ... QUICK BROWN FOX> notes = repository.find(1, 2) # => [<MyNote... QUICK BROWN FOX>, <MyNote ... FAST WHITE DOG>]
When the document with a specific ID isn’t found, a nil
is returned instead of the deserialized object:
notes = repository.find(1, 3, 2) # => [<MyNote ...>, nil, <MyNote ...>]
Handle the missing objects in the application code, or call compact
on the result.
Search¶ ↑
The search
method is used to retrieve objects from the repository by a query string or definition in the Elasticsearch
DSL:
repository.search('fox or dog').to_a # GET http://localhost:9200/notes_development/_doc/_search?q=fox # => [<MyNote ... FOX ...>, <MyNote ... DOG ...>] repository.search(query: { match: { title: 'fox dog' } }).to_a # GET http://localhost:9200/notes_development/_doc/_search # > {"query":{"match":{"title":"fox dog"}}} # => [<MyNote ... FOX ...>, <MyNote ... DOG ...>]
The returned object is an instance of the Elasticsearch::Persistence::Repository::Response::Results
class, which provides access to the results, the full returned response and hits.
results = repository.search(query: { match: { title: 'fox dog' } }) # Iterate over the objects # results.each do |note| puts "* #{note.attributes[:title]}" end # * QUICK BROWN FOX # * FAST WHITE DOG # Iterate over the objects and hits # results.each_with_hit do |note, hit| puts "* #{note.attributes[:title]}, score: #{hit._score}" end # * QUICK BROWN FOX, score: 0.29930896 # * FAST WHITE DOG, score: 0.29930896 # Get total results # results.total # => 2 # Access the raw response as a Hashie::Mash instance. # Note that a Hashie::Mash will only be created if the 'response' method is called on the results. results.response._shards.failed # => 0 # Access the raw response results.raw_response # => {...}
Example Application¶ ↑
An example Sinatra application is available in {examples/notes/application.rb
}, and demonstrates a rich set of features:
-
How to create and configure a custom repository class
-
How to work with a plain Ruby class as the domain object
-
How to integrate the repository with a Sinatra application
-
How to write complex search definitions, including pagination, highlighting and aggregations
-
How to use search results in the application view
The ActiveRecord Pattern¶ ↑
The ActiveRecord pattern has been deprecated as of version 6.0.0 of this gem. Please use the Repository Pattern instead. For more information on migrating 5.x ActiveRecord-based applications to use the Repository Pattern, please see this blog post.
License¶ ↑
This software is licensed under the Apache 2 license, quoted below.
Licensed to Elasticsearch B.V. under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. Elasticsearch B.V. licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.