Ideas and volunteers for SI4T 2.0

Question

We are starting to plan a new release for SI4T - the open source search integration framework for Tridion. As such we are interested to hear any experiences with SI4T (positive and negative), ideas for version 2.0 features, and of course get volunteers onboard. Please answer here or drop a mail to si4towners@gmail.com

UPDATE: Things are now underway after a kickoff meeting. The presentation and recording are linked below. There is a skype chat group to share ideas and discuss work related to SI4T 2.0 so if you want to join in, please mail your skype id to si4towners@gmail.com with some information about what/how you will contribute. This topic remains open for scope definition, so please add them if you have additional ideas/thoughts.

• Presentation: http://prezi.com/xthtwik5wtcn/si4t-20-kickoff/
• Recording: https://sdl.webex.com/sdl/ldr.php?RCID=5364cfd165cab2d5923c3a8577e537bd

Answer 1

I've used SI4T for one completed project and one that's currently in development so I've got a bit of experience with it.

I really like the flexibility of the mappings between component data and the index (in our case Solr) and we've been able to get most of the stuff we need into Solr very easily which has been brilliant.

We've had a few more challenging issues at the storage extension side though so I thought it would be worth feeding back on those while they're fresh in my mind now (and I'll add any more I can think of in the next few days) so that you're aware of them and also to maybe be integrated into v2.0.

The biggest issue we ran into was publishing multimedia components which also had metadata attached. We have a DCP for the MM component which is used elsewhere but allowed us to add in the Si4T TBBs and map fields to the index data. The CM side worked great. However on the Storage Extension side, because the DCP was being stored in the DB and the binary (PDF) was being stored on the filestore, they would be handled with 2 separate SearchIndexer instances and as such would effectively overwrite each other when committed to Solr. ie. we would either get the extracted data from the PDF or the metadata and not the binary's content. We had to implement partial updating in the Storage Extension to solve this issue. It's a bit hard coded and hacky (time constraints) to have provided you with the code to release earlier but I can certainly send over what we did if it helps get a proper resolution in v2.0.

The above issue also raised a potential bug in the SearchIndexProcessor.java code in the triggerIndexing function. The function loops over the actions before setting the SearchIndex and performing the actions. However, it's only outside the for loop that the commit function is called on the SearchIndex object, meaning (as in the above case with the binary and the DCP) only one of these is being committed.

See comments below (I've removed some lines of code to keep it short and to the point!)

public void triggerIndexing(String transactionId)
    {
        if (notificationRegister.containsKey(transactionId))
        {

            SearchIndex s = null; <--DECLARE THE SEARCHINDEX OBJECT

            for (String itemId : actions.keySet())
            {
                BaseIndexData data = actions.get(itemId);

                s = this.searchIndexer.get(data.getStorageId()); <-- INSTANTIATE SEARCHINDEX IN THE LOOP

                processAction(s, actions, itemId); <-- PERFORM ACTIONS ON THE SEARCHINDEX


            } <-- END FOR LOOP



            if (s != null)
            {
                s.commit(pubId); <-- COMMIT THE (LAST) SEARCHINDEX
            }
        }
    }

As you can see, only the SearchIndex object for the storage type of the last action item will ever be committed.

Hope that helps for now. I will add more as I can remember them.

Thanks Mike

Answer 2

'Hello Will,

As you know I'm more then happy to help you with anything you need help with;

Additional, I would like to suggest the following changes.

Change the Unique page ID to another Unique ID when inserting to Solr. This way we can have multiple pages with the same page ID in Solr.

Change so that a page can have multiple entries in Solr.

Suggestion; Change the custom field in xml to customs, and create a predefined field if multiple entries, like custom.

Example;

<customs>
  <custom>Page with content of component A</custom>
  <custom>Page with content of component B</custom>
  <custom>Page with content of component C</custom>
</customs>

The reason behind this is so that a search can be created for multiple tabs, each having there own page url like we discussed earlier =)

Answer 3

One of the things that I would like to see built in is automated build configurations for:

• Multiple Tridion versions
• Multiple Solr versions

The first step in doing this, is by creating a proper build system, for instance by using Maven. This is already well underway. This is also very useful for developers wishing to make changes in the code base, while pulling dependencies from for instance a public Nexus instance, barring of course copyright protected dependencies.

The second step would be to simply have a configuration page where everyone can choose his or her versions and download a set of jar-files for inclusion in the deployer directory.

Answer 4

We've added logic to the projects we've built here that allows a bit more flexibility when using the custom mappings. We added the following notations:

• '@' allows the value to be read from the package

ie. data_type:@dataType would look for dataType in the package and add that. This was really useful for where data would need to be generated using another TBB (such as getting a metadata value from an ancestor SG)

• "" allows non transformed strings to be added to Solr

ie. data_type:"news" would add 'news' to the Solr index field which is really useful when you have a known data type mapping directly to a component template but the data isn't available in the component data - you know it's a news article because it's published using the News CT.

• '#' allows the ID of the item to be indexed, rather than the text value.

ie. For us, this was useful when we needed to index the facet name (from taxonomy) so it's searchable, but we want to facet on the Id of the keyword so that changes in the keyword display name won't break the facets currently published. It allowed us to add mappings like this: category_id:#category+|category_name:category+

I think notations like these have proved really useful for us and would really help with the flexibility of the mappings system already provided.

Answer 5

Another comment I have heard is that there is nothing to help you set up the querying and display of results. While this is in line with the SI4T mission statement, I think it makes sense to have some code packaged as examples, purely to enable you to set up an end-to-end working solution quickly to evaluate SI4T. There are already a couple of NuGet packages for .NET (MVC), but it would be good to have Java and Javascript examples also.

Answer 6

In the beginning we had the ambition to support the following indexing strategies:

1. PUSH (implemented in SI4T 1.0)
2. NOTIFY (simply notify a crawler that there is new content published, and where)
3. FILE (publish the index data to the file system in a data format, such as XML)
4. FILE+NOTIFY (publish the index data to the file system and notify a crawler)

2-4 were not implemented in 1.0, but would be relatively straightforward making the deployment architecture much more flexible. I think we should only do this if there are real use cases out there (I do know of one implementation of SI4T that was extended to do FILE, for example, but I have worked on other projects in the past which used this architecture - its useful if you want to decouple the search engine from your deployment architecture, and has the advantage of being able to re-index without republishing).

Answer 7

Its not something that should necessarily be part of SI4T 2.0, but I want to make sure that there are more indexer examples out there in the public domain. I know that people have worked on Indexers for Elastic Search, FAST and other search engines, so it would be good to encourage people to share these, or to put them into a SI4T repo alongside the Solr one.