Elasticsearch and Tire: adding analysis settings breaks exclude from _source?

Tag: elasticsearch , tire Author: Love_Fang Date: 2012-07-25

I have an object being indexed into Elasticsearch with large attachments, which I don't want stored in _source. Specifying "excludes" in the mapping fixed that (thanks to another question answered here on SO), but when I added custom settings for the filter/analyzer, it's broken the exclude somehow, and I'm wondering if it's a Tire issue or if it simply requires specifying the exclude differently.

Here's the code:

settings analysis: {
  filter: {
    ngram_filter: {
      type: "nGram",
      min_gram: 2,
      max_gram: 12
    }
  },
  index_analyzer: {
    index_ngram_analyzer: {
      type: "custom",
      tokenizer: "standard",
      filter: ["lowercase", "ngram_filter"]
    }
  },
  search_analyzer: {
    search_ngram_analyzer: {
      type: "custom",
      tokenizer: "standard",
      filter: ["standard", "lowercase", "ngram_filter"]
    }
  }
} do
  mapping :_source => { :excludes => ['attachment'] } do
    indexes :id, :type => 'integer'
    [:title, :abstract].each do |attribute|
      indexes attribute, type: 'string', analyzer: 'ngram_analyzer'
    end
    indexes :attachment, :type => 'attachment'
  end
end

The "mapping" line with the ":excludes" was doing the trick until I added the "settings" block, so something about that has caused the exclude to be ignored. Any ideas out there? Thanks in advance!

Best Answer

Try this:

settings :analysis => {
  :filter  => {
    :ngram_filter => {
      :type => "nGram",
      :min_gram => 2,
      :max_gram => 12
    }
  },
  :analyzer => {
    :index_ngram_analyzer => {
      :type  => "custom",
      :tokenizer  => "standard",
      :filter  => ["lowercase", "ngram_filter"]
    },
    :search_ngram_analyzer => {
      :type  => "custom",
      :tokenizer  => "standard",
      :filter  => ["standard", "lowercase", "ngram_filter"]
    }
  }
} do
  mapping :_source => { :excludes => ['attachment'] } do
    indexes :id, :type => 'integer'
    [:title, :abstract].each do |attribute|
      indexes attribute, :type => 'string', :index_analyzer => 'index_ngram_analyzer', :search_analyzer => 'search_ngram_analyzer'
    end
    indexes :attachment, :type => 'attachment'
  end
end

comments:

Ah...thank you. The "mapping" of symbols in Tire to keys in Elasticsearch takes some time to get used to, but this helps clarify it a great deal. Problem solved, thanks again!