Managing Corporate Information Assets with Efficient Enterprise Search & Retrieval

As a practice, Enterprise Search & Retrieval (ESR) has been around in a rudimentary form ever since computers and IT were employed for enterprise needs. However, with the massive explosion of data produced, shared and consumed, ESR has become all-pervasive for small businesses as well as large business houses. Typically, ESR coverage spans organization-wide internal or private search systems rather than using the Internet.

So, what is ESR actually?

ESR can be defined as the use of information retrieval technology for storing, tracking, retrieving, and accessing data within a corporate network. It provides the system to search both unstructured and structured data through search queries. In the standard context, data and information may be stored in various sources such as e-mails, servers, network drives, messaging applications, databases, intranet sites, knowledge management systems, file systems and content management systems.

Benefits of an efficient search solution

Having an ESR system set up within the company helps users to query information and obtain search results in an easy way. With the recent demand for collecting and retrieving large scale information, it has become necessary that ESR also encompass intelligent crawling, clustering, functional categorization, indexing and semantic analysis. Additionally, they also need to ensure compliance with corporate IT security and data confidentiality norms.
An effective search solution can benefit the organization in many ways that eventually lead to increase in productivity (faster results with relevant information) and sales (showing right products based on search criteria).

Types of search

Within the ESR realm, search can be divided into four major categories

1. Parametric search – Is the most basic form of search (think of it as a standard Google search). Here the user will provide search query and all parameters, and the search system will return appropriate results. So if you query “outdoor”, you will get all instances of documents, file names, emails and other resources that have the occurrences of the word “outdoor”. If you refine the parameter, then the search results will be narrowed down accordingly. 
2. Faceted search – This is almost similar to parametric search, with one key difference. Instead of explicitly stating the search parameters before the search, faceted search allows users to discover facets about a basic search result. They can then use these facets as filters to narrow down the results. Taking multiple facets will help users develop a more targeted search query.
3. Federated search – Very often large companies might tap into knowledge repositories that have their own search engine and separate indexing. If you choose that your ESR systems not re-index again with these new sources but rather simply trigger those sources and then return a common results set, this is federated search.
4. Full-text search – This takes up the literal string of the search query and parses it through the engine. The results show occurrence of the search string (either by itself or with other characters). Some of the good full text search solutions include Solr/Lucene, ElasticSearch and Sphinx.

ESR will cater to a wide range of audience within the corporate network. These may fit into one of the below profiles –

1. Those who know what they are searching for and know how to do it
2. Those who know what they are searching for but don’t know how to do it
3. Those who don’t know what they are searching for

The entire search engine implementation needs to take into account all the above types of users to deliver its full potential.

Challenges in ESR

1. Security – All companies have information such as employee payroll, performance reports, sales forecasts, marketing collaterals etc. That needs to be kept away from unauthorized eyes, both within and outside the company.
   Search engines can be easily equipped to ensure that it is guarded against revealing information or data that is not intended for a particular searcher. Forrester’s Sep 2011 report1 on evaluation of 12 ESR vendors shows that security of the system when interacting with people or servers or other repositories, is one of the criteria used for performance evaluation of the system.  
2. Challenges of big data – Exceptionally large data sets need specialized tools, processes and approaches to derive valuable insights and action points. This is one aspect of ESR that is evolving rapidly in the last 4-5 years. There is significant traction in the industry too to capture this opportunity and provide meaningful solutions for companies. The big ticket acquisitions by IBM and Oracle are a case in point. 
3. Single search system – With the line between structured search (where BI comes in handy) and unstructured search (tackled mainly by ESR) beginning to blur, a key challenge is to provide functionalities within each system so that users get exactly what they are looking for. This convergence presents a challenge for some and an opportunity to knowledge management professionals.

ESR Solutions:

Here are few search solutions that can be used to design an ESR system.

1. Open Source – Apache Lucene Solr popularly known as Solr
   Apache Solr 1.4 is a project from the Apache Software Foundation, which provides support for the Apache community of open-source software projects. It is an open-source ESR tool, built on and associated with a range of open-source components. These provide search functionality comparable to proprietary offerings. Sitting on top of the Lucene Java-based search library, Solr provides a REST-like interface to the range of capabilities expected from a search solution. Similar to popular open source projects, Apache projects are collaborative, consensus based, and offer an open software license
2. Proprietary – HP Autonomy, IBM, Google, Microsoft, Oracle

Case study – Solr & MongoDB based search solution for a semiconductor company

CIGNEX has strived to be ahead of the learning curve to adopt and utilize ESR systems that helps add immense value to its clients’ businesses. A good case study can be the work done for one its clients having 15,000 employees worldwide and $5.5 billion turnover. The client needed expert help in transitioning decade old website developed on custom PHP to Liferay based portal. This needed to work as a Digital Marketing Platform with personalized online accounts for customer and supplier collaboration. The new portal with customized search solution leveraging Solr and MongoDB, implemented by CIGNEX provided a rich user experience, faster contextual search and a 360ᵒ parametric search based product view.

The outcome of this new system itself spoke of its success. The client not only saw an online revenue increase of a massive 200% within first 6 months, it also emerged as the #4 most visited technology site in Japan, up by 154 places since 2012. The carefully designed system also helped reduce page load time by a whopping 90% to just 3-4 seconds.

To wrap up

Enterprise Search & Retrieval has emerged as a viable solution to address the need of a user centric system to handle voluminous structured and unstructured data being collected, collated, created and distributed within an enterprise network in the form of corporate information assets. However, CIOs need to get collective buy-in from employees and have proper security features in place in order to ensure that the search systems shows its true prowess with every search query

When Gartner says that ESR has grown 11.7% YoY from 2007 to 2013, it is high time that CIOs and CTOs take note of the power of a well-designed ESR within the company.

1- http://www.cio.com/article/2399619/business-intelligence/how-to-evaluat…
2 - http://www.cmswire.com/cms/information-management/gartner-mq-for-enterp…