ElasticSearch 6 compatibility note

ElasticSearch 6 introduced a huge breaking change regarding indices : they cannot contain more than one document type anymore. Unfortunately, Ovyka BI Indexer uses three different document types in the same index, and uses parent relationships between types to link documents together (users, issues and issues history).

Moving to ES 6 architecture is a complex change for our plugin, which is made even more complex by the fact that Elastic did not provide a technical solution yet to create parent relationships between indices (to replace the existing type to type parent relationships). Hence, adapting to ES 6 is much more difficult that moving from ES 2 to ES 5 was.

The future of the Ovyka BI Indexer plugin will go through at least 2 steps :

• Status
  colour Green title DONE

   first, a Lite ES 6 version will be released, indexing only issues (not users, and not issues history). This will provide a good business value already, and this is the fastest way to provide ES 6 support;

• Status
  title TO DO

   then, we will support indexing all the data we already process in ES 5 (users, issues and issues history) in dedicated indices, but without the link between them. This will allow more BI analysis, but will require a little mapping work on our customers' side;

• Status
  title TO DO

   finally, when we get the information from Elastic, we will restore the full indexing behaviour (with relationships) in the full ES 6 version of our connector.

For more information about removal of types in ElasticSearch 6, please go to the dedicated page at Elastic : https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html

Setup

To install the Ovyka BI Indexer plugin, go to JIRA administration, then go to the Manage Add-ons section. On the add-ons administration page, click the Upload add-on button, as shown in the screenshot below.

Once the plugin is installed, a new menu appears on the administration left-hand menu (you may have to reload the page to see the menu appear).

Administration

The Ovyka BI Indexer - ElasticSearch for Jira (called Ovyka BI Indexer in the rest of this document) administration is separated in four sections :

• ElasticSearch cluster
  • In this administration section, you can configure the plugin so it knows how to connect to your ElasticSearch cluster. This will allow the execution of administrative and indexing tasks.
• Contexts
  • This section allows you to define some indexing contexts, to select which issues from which projects will be indexed in the ElasticSearch cluster.
• Mappings
  • This section allows you to define which issue fields must be indexed in the ElasticSearch index.
• Bulk Indexing
  • This section allows you to index already existing data (issues, comments and history) in your ElasticSearch cluster. Otherwise, the only indexed data will be the data created or modified since Ovyka BI Indexer was installed and configured.
  • You can also reindex your data if you create a new mapping that requires recreating the index in ElasticSearch (some mapping changes in ElasticSearch are not allowed and require that you delete and recreate the index with the new mapping - Bulk indexing allows you to reindex all your data in ElasticSearch if this happens).

Connector setup

Ovyka BI Indexer is only a part of the indexing system. It's role is to integrate into Jira, allows the configuration of what is indexed and how it is indexed, and listen to JIRA events to index any change in configured contexts' issues. When any interaction with the ElasticSearch cluster is required (check cluster status, create or edit index mapping, index data), then Ovyka BI Indexer needs the help of another add-on, called a connector, whose role is to communicate with ElasticSearch. If you didn't download one of the connectors listed at the top of this document, go ahead and download one of them, matching your ElasticSearch version (2 or 5). Once downloaded, install it in JIRA using the Manage add-ons page of JIRA administration, with the same processed described above to install Ovyka BI Indexer.

To know if a connector is installed and active, you can go to the cluster administration of Ovyka BI Indexer. If you don't have a connector installed or if it's disabled, you will see this :

Image Removed

If the connector is installed and active, you will see this (if you are using the connector for ElasticSearch 5.x) :

Image Removed

You can also check in the Manage Add-ons administration page of Jira, and you should see both plugins installed and active, as shown below :

Image Removed

ElasticSearch cluster administration

In this section, you must give enough information to allow the add-on to connect to your ElasticSearch cluster. Informations to provide are :

• the ElasticSearch cluster name;
• the IP and port of cluster nodes;
• (Optional) if you are using SearchGuard to secure your ElasticSearch cluster, you must also provide a keystore and truststore, along with their passowrds, and activate SearchGuard to inform the add-on that it must connect through SearchGuard (more on this below).

Cluster configuration

To configure the cluster, you need to provide one ip/port for one of the cluster nodes. Note that since the data center version, we use the HTTP port (9200 by default), not the transport port anymore.

Once a working node is provided, the plugin will discover the list of nodes and the cluster name.

Cluster nodes

Once the configuration of the node is done, the list of nodes will appear automatically, as seen below.

You can see that the clluster name, its status, number of nodes and the list of its nodes were discovered. 

ElasticLink Index

To index JIRA issues in ElasticSearch, Ovyka BI Indexer needs to have a dedicated index in the ElasticSearch cluster (as many indices can co-exist in a cluster, used by many different applications). This screenshot shows the cluster status area with a correctly configured cluster, but no ElasticLink index configured :

As you can see, there is two options for you to select an index

• you already created an index in your ElasticSearch cluster, to receive JIRA indexed data : you can use the Edit link to enter your index name and use it for indexing.
  
• you do not have an index in your cluster for JIRA data. You can use the Create link to create one for indexed JIRA data.
  

Once configured, the cluster status area should change to this :

The configured index is now shown, and you have now the following options :

• Unset : to remove this index from the add-on configuration (revert to the previous situation, with no index configured). This will not delete the index in ElasticSearch. Once unset, you will be able to either enter the name of another existing index, or create a new one directly from the add-on administration.
• Edit : to change the name of the index immediately from one existing index to another (or just to fix its name if you mistyped the index name). This, again, has no effect on ElasticSearch cluster. This only changes the add-on configuration.

Contexts administration

Ovyka BI Indexer plugin uses the notion of contexts to decide wether an issue must be indexed or not. A context, apart from its name, is defined by two elements :

• a list of projects;
• a list of issue types.

The combination of these two elements will allow the plugin to know, when an issue is created or updated, if it must be indexed in ElasticSearch or not.

To configure one or more contexts, go to the Contexts administration page. When no context is defined, this page will look like this :

To add a context, click the Add context button under the configured contexts table. This will bring up a pop-up window where you'll be able to configure the context.

In this window, only the title is mandatory. If you don't enter any project or issue type, then this context will apply to all projects and all issue types. It is important to know a few things about a context configuration :

• a context title must be unique. If you enter a context title that already exists, an error will be shown and the context will not be saved;
• A context perimeter must be unique. This means that an issue cannot be associated with two contexts. For example, if you configure a context Context1 for All projects and the Bug issue type, you cannot configure another context Context2 for project DEMO and the Bug and Task issue types. Because if you do so, an issue in the DEMO project of type Bug will match Context2 context, of course, but it would also match Context1 (All projects and issue type Bug). When you know more about mappings (explained later in this document), you will understand why this is not possible. Anyway, if you configure a context that would conflict with an existing one, an error will be shown and your context will not be saved.

Below is an example context configuration :

Issue types and projects inputs help you in your selection. Start typing, and a drop down menu containing projects or issue types matching the entered text will appear, as shown below. Just select one of them, or continue typing to refine the proposed choices.

When your context is created, it appears in the list of configured contexts, as shown in the screenshot below :

Of course you can still modify a context, or delete it, by using the Edit or Delete links in front of the context you want to modify or delete. Please note that in some cases, you won't be able to delete a context. This will happen if the context you want to delete is associated with a mapping. More on this below.

Mappings administration

The last administration section to configure is the Mapping section. A mapping defines how issues will be indexed in ElasticSearch. To be more specific, it allows you to choose which fields from issues matching a context (this will be explained later) will be used for ElasticSearch indexation. If a field is indexed, it is searchable in ElasticSearch. If it is not, you won't be able to search issues based on values from this field.

The default mapping administration screen looks like this :

An information note is visible here to inform you that as long as no mapping is provided for the context(s) you defined, no indexation in ElasticSearch will be performed.

Users mapping name

A very important part of the mapping is the name that will be given to your users index. If you are not satisfied with the default name (jira_users) then change it and hit the Update button.

Fields mappings

To add a new mapping, click the Add mapping button under the configured mappings. This will bring up a pop-up window where you can configure your mapping, as shown below :

As for contexts, the mapping title is mandatory, and must be unique. And as for contexts, mapping perimeter must be unique. This means you cannot configure two mapping associated with a common context.

Like in the context configuration window, the contexts selection here offers assistance. Start typing, and existing contexts will appear. Select one of them or continue typing to refine the proposed matching contexts. See it below in action :

As an example, here is a mapping definition, matching the context example created earlier in this document :

Once you hit the Save button, and provided no conflict is detected, your mapping will appear in the table of configured contexts, as shown in the screenshot below :

The mapping will be created in ElasticSearch and synchronized (if everything is correctly configured).

Let's see what are Configuration and Synchronization ?

Mapping configuration

We quickly mentioned earlier in this document that a mapping allows you to choose which fields of an issue will be indexed in ElasticSearch. The mapping configuration is the place where you configure this.

To begin configuring your mapping (or to change its configuration), click the Configure link in front of the mapping you want to configure. This opens a configuration panel under the mappings table, as shown below :

You may be wondering where this fields come from, and why they appear here, and others do not. Ovyka Bi Indexer uses the context(s) you associated with your mapping to discover the maximum common field between all projects and issue types configured in associated contexts. If a field is available in an issue type of a project configured in the context associated with your mapping, it will be listed here. If a field is available in the Task issue type of the project configured in your context but you only configured your context to index Bug issue types, then this field will not appear here.

Another important point to note. The Key field does not appear in the list, because (as explained in the configuration panel) the issue key is always indexed. And by the way, the ElasticSearch document id for the indexed issue is the issue key. This allows you to search by issue key or to get an issue by its key (document id in ElasticSearch).

Here is an example mapping configuration :

Once you configured and saved your configuration, it should synchronize (see below), but if it fails, it will be marked as Configured (saved in plugin configuration), but not synchronized (mapping was not configured in ElasticSearch index). If everything is fine, you should see tour mapping configured and synchronized.

Mapping synchronization

Once a mapping is configured, you have to give this configuration to ElasticSearch, so it knows how to index the issue the plugin will send for indexation. This process is called synchronization. When you hit the Synchronize link of a mapping, Ovyka BI Indexer will convert your configuration to an ElasticSearch mapping and create (or update) a type in the ElasticSearch index. The name of the ElasticSearch type corresponding to your mapping is based on the mapping name : Ovyka BI Indexer will convert the mapping title to lower case, and replace all spaces with underscores. As an example, for the mapping created in the screenshot above (Demo Issues), the ElasticSearch type will be named demo_issues.

If the synchronization is successful, an information message will be shown, as shown below :

And the mapping in the mappings table will be displayed as synchronized, as you can see in the screenshot below :

Please note that as soon as you change the mapping configuration, the mapping will be re-synchronized. Make sure it appears as Synchronized in the table, or click the Synchronize link.If your changes are not synced, then they will not be used by ElasticSearch.

Bulk indexing

When installing Ovyka BI Indexer in a JIRA instance that already contains data, or if you deleted and recreated the dedicated ElasticSearch index, you may want to index (or reindex) all existing data from your JIRA instance into your ElasticSearch index. You can achieve this in the Bulk Indexing administration screen. The screenshot below shows the Bulk indexing administration.

The buttons here allow you to :

• Purge : Delete all existing data from your index
• Reindex All : Start indexing all existing JIRA data into the ElasticSearch index (according to configured contexts and mappings)
  • This is useful on an instance where you just installed Ovyka BI Indexer for the first time;
  • It will work even if your data is indexed, but will not delete data from field that where indexed before but not indexed anymore after a mapping change. You'll need to purge first to delete this data.
• Purge & Reindex All : First delete all existing data in ElasticSearch index then reindex all JIRA data into this index.
  • Cleans the ElasticSearch index from all existing data, then index all JIRA data into the ElasticSearch index.

Issue indexation

Issue indexation is triggered :

• when editing or creating an issue, adding or editing a comment provided the issue matches one of the configured contexts;
• when manually launching the bulk indexation process, which will index (or reindex) all existing issues.

Types hierarchy

Ovyka BI Indexer creates three  types in your index :

• users;
• issues;
• history.

Users

This type will hold a document per user in your JIRA instance.

Issues

This type will hold a document for each issue in your JIRA instance (provided it matches a context and a mapping). Issues documents are linked to a parent document in the Users type, their reporter user.

History

All change in your issues is indexed in this type. There will be multiple documents in this type for each issue, tracking all changes that occured to this issue. Of course all history documents are linked to their parent issue's document.