The search continues: A history of search's unsatisfactory progress

It is clear that search has not changed much in the past 20 years. Back in the 1990s, enterprise search first included indexing multiple, heterogeneous data sets into a single search experience, with full document-level security, using search engines and this pretty much sums up the situation today.

A survey conducted by SearchYourCloud, a search and security company, revealed that a third of respondents spend between five and 25 minutes searching every time they want to find a document, while only one in five searches is correct the first time. The search for corporate information is eating into workplace productivity. Only 20 per cent of respondents reported first time successful searches. Other key findings from the survey include that it takes workers up to eight searches to find the right document and information, according to 80 per cent of respondents.

Verity began offering search in the late 1980s until it was acquired by Autonomy in 2006. Previously, Verity provided unified results from multiple, simultaneous searches from the desktop to the enterprise. While Verity took a semantic approach to search, Autonomy took a statistical approach to understand statistical relationships between terms. In 2011, HP acquired Autonomy to bolster its search and analytics business. Also in 2011, Oracle, which launched Oracle Secure Search in 2006, acquired Endeca. In order to improve SharePoint search capabilities, Microsoft acquired FAST Search and Development.

While it may seem like the search industry grew in this time period with these new companies emerging, it is actually the opposite. Rather, Autonomy and Microsoft came along and made search more about consultancy and less about usability or actual results. It took what was a vital part of the search world and combined services to enhance it for general enterprise, not make it easier to search your information.

Nevertheless, in the past five years, the demand for a secure search that delivers results quickly has grown, in part, due to Big Data, mobility and cloud services. Big Data encompasses the massive amounts of data that is stored in databases, spreadsheets, emails, reports, etc. and generally needs to be searched separately. With mobile and cloud services, users and their devices are more dispersed with important files stored on numerous devices, on-premises and in the cloud. The problem is made even worse because most data is collected into large repositories, which are slower and more complicated to search, which results in companies’ having a big “pile” of data, whether in a database, series of Excels or as stored photos, emails and other files, that are unusable and consequently reduce productivity.

With the advent of federated search, the ability to search across multiple repositories has improved. Moreover, with federated de-duplicated results, users do not receive thousands of irrelevant documents or emails. Users can simultaneously search across applications. It is best to take a non-repository processing approach and keep the existing data silos separate. A large repository can be kludgy with inherent security risks and to combine multiple silos may create problems in reconciling different processing power and security levels.

To find needed information, an enterprise search tool must deliver the exact results in a timely manner and not a boatload of assorted data that does not deliver the data the person actually needs. Unlike a Google search that finds what is considered to be most relevant based on visits and cookies and the more results the better, enterprise search should deliver information that is relevant to only that task at hand - payroll, results, competitive information, etc. - be it in Excel, SharePoint or HANA database.

Furthermore, because this type of information is typically private data, the delivered results need to be available to only those who need access while not allowing those who wish to use the information without permission. Consequently, it is important to determine which question you are asking the data to answer. Rapid response is also key, which is another reason that search needs to find the data so you can quickly act on it. Whether it is to solve a client issue, approve overtime and/or commissions, present information to the board of directors, or improve the sales process, it’s critical to have instant access to accurate search results to keep enterprise productivity.

The good news is that search is evolving, albeit slowly. Developers of enterprise search are now coming to realise that it must:

  • Bring needed answers and/or files and not merely a list of results.
  • Learn from analytics and past searches
  • Be able to search seamlessly across multiple repositories
  • Deliver results blindingly fast

While search has not improved much in the past 20 years, there are new federated searches that can securely find the right document at the right time. These new types of search also add security to protect privacy of files as they traverse networks.

Things are looking up in the search world, so stay tuned on the journey.

Simon Bain, company founder, CEO and chief architect of SearchYourCloud’s software solutions.

Image Credit: alphaspirit / Shutterstock