14 Industry Experts Predict the Future of Search

Last updated 11:31h 17 Aug, 2015&nbspin Enterprise Search by Cbhan Phillipson

Effective enterprise search represents one of the most challenging areas in business today. The whole area of search has been revolutionized by Google. Employees now expect to be able to locate relevant data as easily as they navigate the web through Google and other search engines in their private lives. When this ease of search is not replicated in our professional lives it can be quite frustrating. As we create more content than ever before, the importance of effective search across the enterprise continues to grow.

There is a popular misconception that search emerged in conjunction with the Internet. In fact, enterprise search dates as far back as the 1960s when IBM developed an early internal search engine. What the advent of the internet and Google did bring about was a new set of expectations surrounding enterprise search. Google set the standard and enterprise search is expected to follow suit.

We wanted to gain a clearer understanding of current state of the enterprise search industry. Given the steady evolution of enterprise search, we also wanted to gain some insight into what the future may hold. To do so, we gathered a select number of industry experts and asked two simple questions:

1. What is your assessment of todays enterprise search industry?2. What do you think the future of search will look like?

Weve compiled their expert insights into this comprehensive assessment of the enterprise search industry. With the caliber of experts who partook, were sure this interview will help you evaluate the state of search at your own company, and even improve overall productivity as a result.

Meet Our Panel of Search Experts:Steven Nicolaou, Principal Consultant, Microsoft

Steven Nicolaou has been an enterprise search solution architect since 2003 as part of FAST Search. He joined Microsoft upon its acquisition in 2008 and now architects global solutions with SharePoint Search. Customer projects have included Dell, Best Buy, BP, Merck, Pfizer, Barnes & Noble, Marriott, Xerox, Goldman Sachs, McGraw Hill, NOV, CambridgeSoft and the US Federal Government.

What is your assessment of todays enterprise search industry?

Enterprise Search today is a developing industry. Until recently, much of the core technology remained unchanged since the 1970s and innovation was fairly limited to niche markets. The general purpose enterprise search offerings were fairly similar in technology and scope.

The past decade however, has seen feverish growth in the field due to a variety of forces, some of which include:

Big DataGoogleCloud Computing

This fact was recognized in the second half of the 2000s, when the niche http://www.toodledo.com/public/td55fb6d6bde47e/0/1578439/News.html search companies were absorbed in rapid succession by the software behemoths:

Microsoft acquired FAST Search in 2008Adobe acquired Mercado in 2009Dassault Systmes acquired Exalead in 2010Hewlett Packard acquired Autonomy in 2011Oracle acquired Endeca in 2011IBM acquired Vivisimo in 2012

With these mighty powers now driving innovation, the field is poised for a new era of visibility and innovation.

Lets review the impact of these forces:

Big Data

As the cost of storage decreases exponentially, companies are becoming increasingly liberal in their storage choices. They are putting less emphasis on pruning and curating data and more on using search engines as tools to wade through the ever increasing volumes, in part because it has become cheaper to do the latter.

This not only puts demand on search engines to handle larger volumes of content, but it also increases the demand for handling poor quality data with duplicates, corrupted files and files with missing or incorrect metadata becoming a fact of life. The burden of data quality management is falling increasingly away from content management and onto enterprise search.

Google

Google deserves the credit for simultaneously popularizing search and elevating user expectations. It used to be common for business analysts and engineers to design the search experience and then train employees to use it. Search manuals were often part of a users training pack. Google awakened a new generation of enterprise search users, who, being Google users themselves, demanded the same power and simplicity from their internal search experience. Why cant it be like Google? became a common response to complex or counterintuitive functionality. Because thats how Google does it became sufficient business justification for enterprise requirements. Google has raised expectations of quality in the field of enterprise search and introduced a competitive force from outside the field through the voices of its users.

Cloud Computing

Management of infrastructure, platforms and storage is increasingly outsourced into the cloud as the burgeoning market makes it more cost effective for companies to do so. Search as a cloud service, whether as its own PaaS or as part of a SaaS model, introduces new challenges that are common to all cloud services: multi-tenancy, on-demand scaling and availability to name a few.

What do you think the future of search will look like?

With all the major software houses now directing serious R&D towards enterprise search, the future will bring shorter innovation cycles, continuous user experience improvements, deeper integration with first and third party applications and more ETL-like functionality to handle poor quality content.

User experience is a broad topic in itself, with active trends including:

Richer information about the user to determine context, such as their business context, social context, mobile device sensors, location, speech recognition, preferences and historical usageAdvances in visualization such as HTML 5Natural language processing as in the trends seen with Wolfram Alpha and smart phone digital assistants, such as Apples Siri, Microsofts Cortana and Google NowRicher results that look less like a page of links and more like answers to questionsElements of Knowledge Management that add meaning to queries and results

Enterprise search products will become increasingly and more deeply integrated with existing platforms, allowing more types of content to be searchable and in more meaningful ways. It will also become increasingly commoditized, making it less of a dark art and more of a platform for discovery and analysis.

John Challis, CEO & CTO, Concept Searching

John Challis is CEO and CTO of Concept Searching. He is an experienced entrepreneur having had success with several previous ventures involving the management of unstructured data. In 1990 he founded Imagesolve International which quickly became the UKs leading supplier of Document Image Processing and Workflow products. John then launched ImageFirst Office for BancTec in the USA in 1995 closing over $5m new business in the first 12 months. Prior to Concept Searching he was CTO at Smartlogik; the company behind the worlds first Probabilistic search engine. He is the originator of the companys compound term processing technology and is the driving force behind the product strategy.

What is your assessment of todays enterprise search industry?

At the height of the dot-com boom (15 years ago) it seemed likely that enterprise search was ready to move beyond simple keyword and Boolean searching being replaced by more sophisticated techniques developed by companies such as: Autonomy, Verity, Convera and FAST. These products offered a variety of new algorithms: automatic query expansion, compound term processing, semantic networks, linguistic analysis, concept searching, feature extraction, custom term weighting, etc. What actually happened was that each of these companies was acquired and their products either killed off by their new owners (Verity, Convera, FAST) or otherwise fatally injured (Autonomy). Today, enterprise search is dominated by Microsoft and Google with other leading vendors including: Oracle, Solr/Lucene and ISYS. None of the current offerings moves significantly beyond keyword and Boolean searching. In my opinion the enterprise search market today has little appetite for more sophisticated products the likes of which we have seen come and go in recent years.

What do you think the future of search will look like?

The future of enterprise search seems destined to continue with simple keyword and Boolean searching, augmented by faceted navigation based on metadata. The main driver for this is the World Wide Web. Virtually every e-commerce web site today offers guided navigation based on metadata. When you enter a simple text query into Amazon or eBay, or virtually any other shopping site, you see filters for vendor, price range, colour, size, etc. This ubiquitous model now appears in most of the leading enterprise search products and users immediately understand how a simple text query can quickly be focussed to a specific domain by clicking on a metadata filter. This updated search model is increasing demand for auto-classification products which can generate descriptive metadata automatically based on an analysis of the documents unstructured content

Seth Redmore, VP Product Management and Marketing, Lexalytics, Inc

Seth Redmore is VP of Product Management and Marketing at Lexalytics, Inc. Seth has over 15 years of experience in product management and 10 years of experience in text analytics from the perspective of a user as well as a vendor. Seth has worked in a number of executive roles at both hardware and software companies, including co-founding Netiverse (who built a high-speed server load balancing system) which was bought by Cisco Systems in 2000. Seth has a degree in Chemistry from Carnegie Mellon University.

What is your assessment of todays enterprise search industry?

Open source software has made significant inroads, displacing many of the traditional search vendors. Lucene and its supporting companies like LucidWorks provide solid search functionality at a hard-to-beat price. Where vendors are seeing success is in four main areas:

Providing functionality beyond typical search extending to facets, true knowledge management, multimedia search, and other functionality.Focusing on vertical-specific applications like fraud and supply-chain management.Working with larger, more conservative enterprises.Providing a SaaS, one-stop-shop for zero (or low) touch functionality.

At Lexalytics, we provide a text mining engine thats used by a number of search partners like Coveo, Playence, and Oracle to add additional metadata to their search additional intelligence around just what do those words actually mean? In other words, boosting the value of search by providing more information into the index. This enables other applications, and helps search be smarter useful differentiation.

What do you think the future of search will look like?

A few major factors are going to drive the industry going forward:

Open source will continue to get better and drive out inefficiency in the market closed source better add value.More, better information about the searcher: location awareness, profile sharing, time dependence, deeper understanding of the context and content of the search. With this information, you can provide better, more relevant results. Much of this is in place, but its not all talking to each other yet, and a lot of deep learning contextual understanding is just coming online.Lower tolerance for hassle: people expect search to just work not understanding that it can be just as complicated as any other major IT initiative. By having low-touch solutions, SaaS providers will make major progress in the small/medium business world.Search all the things!: Integrated understanding of objects, video, speech, as well as traditional semantic sources like text will combine together better into a whole that allows for information retrieval no matter what the format or question.Why should you have to ask first?: Search has been traditionally driven by the searcher (duh), but interesting projects that allow for integrated understanding of where/what the user is doing allows for proactive intervention. Why wait for the slow brain to catch up to the fast machine, when the machine can push out what the user needs right then? Yes, this is functionality youre starting to see with Google and other companies, but there are certainly interesting use cases for the larger enterprises, particularly internally to start, and then helping customers as they grow in their relationship with the company.Jim Jackson, Product Manager, MaxxCAT

Jim Jackson is the Product Manager for the MaxxCAT EX-Series of enterprise search appliances. Prior to taking on the product management role, Jim was on the development team that built the original MaxxCAT search kernel and has also been involved in numerous implementations of MaxxCAT into governmental, educational and industrial applications. Jim has a Masters in Computer Science from Cornell University and spent the early part of his career in robotics and unix development.

What is your assessment of todays enterprise search industry?

The volumes of data to be searched continue to increase exponentially, but the information content is not increasing nearly as rapidly, hence, spam is growing. Users do not want to find 5 exact copies of the same information, they want to find an answer to a question. So some of the older legacy search platforms are going to need to completely re-engineer their products to be able to index data sets that are growing exponentially, without increasing the amount of spam in results.

Too many search vendors talk about search, but that is cumbersome. Users, human and machine, do not want search results, or irrelevant search results, they want information. They do not want to arbitrate between a competing list of 10 SERPS to find what they want.

In fact, search is becoming an integral part of many vertical software applications and what is becoming more and more important is the ability of a search solution to index fast amounts of disparate data from many sources, and to make sense of it and present information, not results lists to users. Buzzwords like federated search and enterprise search place too much focus on search, and not enough on getting the right information, transparently and rapidly to the consumer.

What do you think the future of search will look like?

Search will continue to become more implicit, connecting users to knowledge transparently. Users do not want to search, they want to get information. We are trying to collapse the time to information, and this is not just about extremely fast search of vast amounts of data. It is also about not wasting time Enterprise Search searching and presenting irrelevant information, and about creating results that are fine-grained. People are accustomed to doing a search and getting say 10 best results, each one a document, and then sifting through the documents. We are trying to improve that by returning finer grained results that are not documents, but the exact sentence, the exact spreadsheet cell, or exact information the user is looking for.

Another area for future development is machine to machine consumption of information and sharing. Many consumers of MaxxCAT search results are not humans, but other machines that use MaxxCATs crawling, storage and retrieval capabilities as a black box that simplifies the inclusion of powerful contextual search in other applications. The MaxxCAT 5.0 was a big step to the future for us, the appliance now understands micro tagging, and has much more positional and contextual information in the index to help users zero in on the answer, not on a list of possible answers that show up in a SERP.

To facilitate our vision of implicit, contextual search, MaxxCAT is continuing to enhance our API to allow people and machines to leverage the tremendous search and performance capabilities of our information platform so that solutions can be built that connect people to the information they want. In parallel with enhanced API capabilities, MaxxCAT is also building elastic scaling and cloud capabilities into every device so that we can continue to scale data sets without giving up reliability, relevance or performance.

David Murgatroyd, VP Engineering, Basis Technology

Dave joined Basis Technology in 2005. He leads the engineering team responsible for text analytics including existing products and new technology initiatives. He has been building natural language processing systems since 1998, including positions at Unveil Technologies, Zoesis, Wildfire Communications, and iConverse. He has a B.S. in computer science and a B.A. in computational and applied mathematics from Rice University and a Masters degree in computer speech and language processing from Cambridge University, U.K.

What is your assessment of todays enterprise search industry?

The industry continues to grow, spurred on by both commercial needs and pressing governmental security challenges. The global enterprise search market reaped revenues of more than $1.47 billion in 2012. That figure is forecast to be $4.68 billion by 2019. The investment in search is motivating a growing focus on search quality. Search providers are increasingly applying advanced analytics of text and other media so their users desires are more deeply satisfied through relevant search results. Along with better quality results, search providers are pursuing broader results from around the world. They may be motivated by commercial expansion to global markets or governmental pursuit of geopolitical challenges. That global content has its own challenges, especially linguistic ones, that are not addressed out-of-the-box by the most commonly deployed search engine platforms.

What do you think the future of search will look like?

Search will be increasingly entity-centric and collaborative. Imagine the director of an advertising agency who wants to take her teams temperature by asking Which of our clients are we discussing most across our internal communication platforms? She wants mentions of Phillips 66 and Philips Gas grouped together while keeping those separate from mentions of the light-bulb-making Philips or even of Philip Morris. If she has staff in Hong Kong she also wants to include mentions of 66. She may want to see an entity card along with her results giving her summary information on the client such as their account manager or year-to-date spending. This is entity-centric or knowledge-graph-based search resolving the different ways people, places, organizations, and other entities appear so that users can focus on those key players rather than be caught up in finding just the right keywords.

Once our advertising director sees that Phillips 66 is the most discussed client, she may want to try to understand why theyre receiving so much attention by asking What are our next milestones for them? Understanding who she means by them is an example of collaborative or conversational search, where the results of one interaction are used as context for the next. Collaborative search enables a virtuous cycle where the users engagement with a search platform honing in on items of interest and discarding the misfires allows the computer to learn from the users actions and present the next results with greater awareness of what is being sought.

These two trends reinforce and complement one another. The user and the system collaborate best when they do so around a shared inventory of real-world entities. Analyzing those sometimes ambiguous entities accurately is best done with the added information provide by rich collaboration.

dtSearch

dtSearch took a consensus-based approach to this group interview, offering an enterprise-wide assessment instead of an individual opinion. The Smart Choice for Text Retrieval since 1991, dtSearch offers enterprise and developer text retrieval along with document filters encompassing a broad spectrum of data formats. You can find hundreds of reviews and developer case studies, as well as fully-functional evaluation versions, at www.dtsearch.com.

What is your assessment of todays enterprise search industry?

The expectation of everywhere search defines todays enterprise search. Enterprise users expect instant concurrent searching of all content-based data applications. And they expect comprehensive, instant concurrent searching across all data repositories at once.

Enterprise users expect instant concurrent search access to traditional databases, like XML, XBASE, CSV, MS Access, Oracle and MS SQL, along with the full-text of BLOB data. Enterprise users expect similar search access to directories with MS Office files, PDF, compression and other Office formats.

Users today also expect instant searching of public Internet and private-secure Intranet data, spanning both static content like HTML, XML/XSL and PDF, and dynamic content like SharePoint, PHP or ASP.NET. And users expect instant concurrent search access to email messages and multilayered nested attachments.

A further expectation is that everywhere search will dig deep. Suppose an email message has a ZIP or RAR attachment consisting of a PDF and an MS Word document, where the latter embeds an Excel file which in turn embeds a PowerPoint document. The expectation is that everywhere search will encompass the deepest levels of this recursively-embedded structure.

Finally, the expectation is that everywhere search will extend far beyond basic Boolean, phrase and proximity features. Instead, the expectation is that instant, multithreaded concurrent everywhere search will encompass everything from credit card to fuzzy and faceted searching. And of course, everywhere searching should return all search results with highlighted hits.

What do you think the future of search will look like?

The future of search involves whatever cutting-edge implementations independent programmers develop using extensible APIs.

Using the dtSearch Engine developer APIs as an example, CodeProject has one recent article describing lightning quick full-text searches with dtSearch across an MS Azure SQL database.

Another article involves deploying dtSearch with an Entity Framework dataset and faceted search navigation. And still another CodeProject article combines Telerik user-interface components with the dtSearch Engine developer APIs.

You can find links to all articles at www.dtsearch.com/contact.html.

Nick de Toustain, Director of Sales, LTU technologies

Nick De Toustain serves as Director of Sales at LTU technologies, a 15-year provider of visual search & image recognition tools. LTUs clients include companies doing social media monitoring, brand protection, mobile visual search, and anyone needing to find a specific image within a massive set of images. Prior to LTU technologies, Nick was in various sales and business development roles at T3Media, Getty Images, and Valtech. He lives in NJ with his family and aspires to be a pro tennis player.

What is your assessment of todays enterprise search industry?

Clearly theres value to be had in being able to sort through a companys disparate data sources and making them accessible so as to deliver actionable business intelligence. The questions for an enterprise search provider then become: how easy is you solution to implement? What data sources does it work on? How are the search results presented? Can it do visual search in addition to text search? And of course: at what price point? The companies optimizing these criteria will ultimately win.

What do you think the future of search will look like?

In terms of visual search what LTU specializes in the future is looking bright. Whether its web sites or social media, everything is becoming increasingly visual. Theres a corresponding need to make sense of all that online imagery, which is where image recognition technology comes in. Future enterprise search tools will need to include image recognition capability to keep up with the massive amount of imagery being tweeted and posted every second.

Otis Gospodnetic, CEO, Sematext Group, Inc.

Otis Gospodneti is the founder of Sematext Group, Inc., a globally distributed organization of Search and Big Data experts who build innovative performance monitoring (SPM), log analytics (Logsene) and search analytics (SSA) products, provide exceptional consulting services, and offer 24/7 Solr and Elasticsearch production support. Otis is a co-author of Lucene in Action and has been involved with Lucene since 2000 and Solr since 2006. He is also a member of the Lucene Project Management Committee as well as an Apache Software Foundation member.

What is your assessment of todays enterprise search industry?

In the search context the phrase enterprise search often means two different things: search of one enterprises content (e.g. email, file servers, etc.) or large scale search. At Sematext we help clients with both of these problems but over the years weve worked on so many large scale search and data processing problems I prefer talking about that more, so let me address that first. The core search functionality that we use today has been around for decades, but we still see it evolving and improving. Users are pushing more data into search engines, asking them to handle more and more queries and answer harder questions. Search engines are no longer just full-text search engines. They are being used as key value stores, databases, NoSQL engines, backends for BI front ends, etc. For example, just today one of our clients a large and well known software vendor asked our Elasticsearch Consultants whether using Elasticsearch as a non-search database. Sooner or later every non-search data store, be it a relational database, columnar database, graph database, or needs to provide search functionality. Some enterprises are realizing this early and switching to using search engines from the get go, thus eliminating the need to bolt search functionality on non-search data stores, avoiding the need to maintain multiple copies of data and keep them in sync, etc. In other words, search engines, at least the open-source ones Solr and Elasticsearch are increasingly being used to solve Big Data Problems.

As for enterprise search in the search of one enterprises content sense, I havent seen a ton of progress or exciting development over the last few years. Ive attended and given talks at various (enterprise) search conferences and whenever there is talk or search within the enterprise I keep hearing the same old topics and problems over and over federated search, silos vs. no silos, Sharepoint, ACLs I typically get a headache from hearing the word Sharepoint by the end of each such conference, which is telling.

The Cloud adoption is going up and organizations are increasingly running their search applications in the Cloud. Id say more than 50% of our Solr/Elasticsearch engagements are with clients who run Solr/Elasticsearch in the Cloud typically AWS.

In addition to organizations running their search apps in the Cloud, say on their EC2 instances, a number of companies are offering Search in the Cloud services, freeing organizations not only from dealing with hardware, but also freeing them from dealing with search software. There are a number of providers in this space offering what is basically hosted Solr or hosted Elasticsearch. Even AWS added CloudSearch recently to their list of services. We havent seen a ton of adoption of these services yet, but Im sure we will see this in the coming year or two.

What do you think the future of search will look like?

Search engines like Solr and Elasticsearch are providing more and more functionality that is not really about pure search any more. Lucene, a venerable open-source Java search library that both Solr and Elasticsearch rely on is over 15 years old, but is still going through rapid evolution because of these new workloads. Well see full-text search embedded in more applications and devices. Well see the line between non-search and search software and servers blurred even more. I suspect we may see people putting query languages with familiar syntax, such as SQL, on top of search engines to enable people to write powerful queries more easily while hiding the original query syntax people typically use with search engines today. We have actually done this for one of our products at Sematext. I suspect we will also see more relational data in search, thus enabling search software to be used for what relational databases have been used so far.

Edward Ross, Solutions Architect, Exorbyte

Edward Ross worked at exorbyte in search engine product development and management for 13 years. He enjoys the challenge of bringing cutting edge algorithms to bear on customer problems in an easy-to-use product. Exorbyte offers market-leading search and match solutions. Identity resolution needs to be intelligent, accurate & fast. No exceptions. No restrictions. exorbyte can do just that with MatchMaker, the leading error-tolerant search & match platform for huge master data volumes. The multiple award-winning software firms technology thinks, searches & finds like a human but dramatically faster, in much more complex configurations & with no serious data restriction using keys or similar methods. Available on-premises & in the cloud.Federal authorities, insurance agencies, ICT firms and more use exorbyte identity resolution software in diverse, data-intensive business processes such as input management, enterprise search & data quality. All due to easy customization & integration. For demonstrably better productivity.

What is your assessment of todays enterprise search industry?

Information retrieval remains a vital IT function. Also customers increasingly understand that accessing data-sources like address- and product data often requires a different solution than classical full-text enterprise search solutions can offer.

What do you think the future of search will look like?

I think the future of search will include more semantic understanding of both content and queries. And there will be more differentiation among vendors. For example at exorbyte we are focussed on searching in structured master data people, products and places, and our ability to query this data without use of restrictive match-keys for both lexicographical and semantic similarity is globally unique.

In addition I see search engines handling more business process automation tasks based on the search index. One example is that you can query the search engine with a complete customer-letter, and the index will relate this unstructured content to structured content. In this example: it will return which customer the letter concerns. Functionality like this helps for example large insurance companies to automate their inbound mail handling processes. An example for this can be found here.

Jordi Prats, CTO, Inbenta

Jordi Prats, educated at UPC in Barcelona, has developed a professional career at Inbenta, a company specialized in Artificial Intelligence and Natural Language Processing. Jordi currently is CTO at Inbenta and has been directly involved for years in the research and development of Inbentas Natural Language Processing technology.

What is your assessment of todays enterprise search industry?

Enterprise Search should be considered a critical piece of software since the efficiency of companies procedures and employees might depend on it, but unfortunately it is often underestimated (Enterprise Content Management Systems are given more priority to the detriment of Enterprise Search, although both components ought to be considered equally important).

Intranets and document repositories tend to gather huge amounts of information and being able to provide a service capable of generating structured indexations from multiple and diverse sources of information (a.k.a collections) as well as targeting multiple audiences through a unique platform is a key item to convert any organization into an efficient machine.

Nowadays, the market is flooded with solutions that try to efficiently tackle Enterprise Search (sometimes Enterprise CMS and Search at the same time), but more often than not companies end up facing costly integrations which only provide partially effective results.

In short, an efficient Enterprise Search system should be a versatile search tool, capable of understanding complex search queries, compartmentalizing information to target multiple audiences and being able to prioritize through different types of sources.

What do you think the future of search will look like?

The need to efficiently browse through huge universes of information (involving Enterprise Search, Web Search or Internet Search itself) gets day by day more important, and search engines are the main gate to information. Therefore, search engines in general, are experiencing big improvements lately, leaving behind the rather obsolete keyword-based search in the benefit of complex systems taking into consideration semantics and natural language processing.

Even though search solutions are currently taking huge steps evolving into smarter systems, users are often left with huge lists of results which require time to browse and filter through and sometimes poor relevancy algorithms could even make search boxes an unfriendly element, while they should be the most efficient door to access information.

In my opinion, on one hand the future of search goes through natural language processing (best understanding of user intention, even evolving into rather interactive systems such as the commonly known virtual assistants, which may allow users to accurately define the real search scope by means of taking a few and simple extra steps before launching a search query), while on the other hand itll entail the capability of providing advanced information analysis during indexation time (best understanding of the sources of information contemplated, for instance grouping information from different sources by meaning and thus granting the ability to bundle together pieces of information from different sources into a single search result). This idea would consist of interacting with a computer in a similar way in which we would interact with another human (mimicking http://www.zdnet.com/article/what-is-big-data/ how humans process information when reading or learning in order to improve indexation processes, mimicking how humans process information when listening in order to understand complex search queries and finally, mimicking how humans structure knowledge in order to build complex answers, so search results would rather become search answers built from multiple sources).

Nonetheless, the future of search still remains an enigma and internet users will gradually lead search research and development towards the most effective ways to find information

John Felahi, Chief Strategy Officer, Content Analyst Company

Mr. Felahi is the Chief Strategy Officer for Content Analyst Company, LLC where he is responsible for the company product strategy and vision. Prior to joining the Content Analyst Company team, he was a Principal Product Planner at Microsoft Corporation, where he was part of a small team responsible for setting the Office product divisions long term product direction and strategy. Mr. Felahi has held executive and senior positions at a number of companies in his 20+ years in the software industry.

What is your assessment of todays enterprise search industry?

There are two perspectives on this: First, what are people mostly using today, and second, what is the industry doing to improve the relevancy of the results to the people with the questions?

For most people, search is a simple word or two in a search box, or voice command, to get back results to three basic actions:

I know what I am looking for. Find it for me. This action is typically used with major search engines (Google, Bing, etc.) and on personal devices (phones, tablets, laptops, etc.), when people dont know the URL to a website, where a document is located, or someones phone number, so they use this search method to find it quickly.I know the parameters of what I am looking for. Help me focus on the right set of content/items. This is popular in online shopping experiences. For example, I am looking for a new TV under $2,000, but at least 60, with internet connectivity, etc.; show me the ones that match my needs and desires.Help me find content on this topic so I can learn more or see what is available. In this scenario, users are trying to learn something they may only have a general idea about, or are not 100% sure what is available to them.

Results from traditional keyword search in the third scenario can be hit or miss. Search can be highly relevant by leveraging dictionaries, synonym lists, search logs, or context of the requester (any information on past searches, IP address, etc.) to improve relevancy. Yet with only a few words for the search engine to go on, the results can be useless and frustrating because they are not relevant to the query. Or if they find a document that is interesting and they wish to read more similar documents, the search engines do a poor job of finding conceptually similar content since they are relying on metadata and unique words in the document to search for other documents.

The enterprise search industry is working harder to leverage more data (search logs, user context, classification metadata, etc.), i.e., its own version of big data, to improve the relevancy of results and reduce false positives and false negatives (the frustrating part of search results). Two other areas where the search industry has made significant progress to improve the relevancy of search results are:

machine learning, to identify relationships between terms/words that the pre-defined dictionaries and other word/grammar rules based tools miss, andvisualization to help see how content is related to the initial question/query. At the end of the day we are trying to make the machine have the knowledge model of a human so the results are relevant and understandable.

What do you think the future of search will look like?

Look like is a good way to think about the future of search: It needs to be visual. Search is following the path of business intelligence (BI) in terms of visualization, but hopefully search gets there at a much faster pace. The hardware environments, machine learning systems and contextual understanding of the requester will only take us so far. So we now need to become more visual with the results since the volumes of content are growing geometrically.

For the first use case above, where a requester looking for a specific item or piece of content, visualization may not be as important. But for the other two approaches above, visualizations can help the requester see and learn about other relevant factors (metadata items like category, date, author, special pricing or requirements, etc.) that can influence what the most relevant result(s) are at the time. And visualization can also help gain insights that are not as obvious with traditional text-only search results.

We also predict that digital systems in the near future will understand human communication within the context of our needs. This means the computer almost has to be better at understanding the voice of the author relative to the requesters question or query. Traditional search leveraging advance lexical based (word list- and rules-based NLP) technologies can misunderstand, misrepresent, or just completely miss important conceptually relevant content.

Machine learning technology can understand how terms (words, abbreviations, and code words for those in the intelligence community and conspiracy theorists) are being used by the author given the use in the document and other documents processed so relevant content is not missed or miscategorized. Social media where the grammar and loose use of language and new made up words, are happening daily and make it almost impossible for lexical approaches to keep up.

Visualizations are only as good as what are fed to them so we need to work on both improved relevancy techniques and the way to show what the system has found. I think a generation of people believes the computer should respond like HAL 9000, and all the other visions in movies of how computers should interact with people. Search is the beginning of the dialogue with HAL 9000, but the question for another day is what is the computer doing to be like HAL 9000 with what it knows about the requester and the content?

Dixon Jones, Marketing Director, Majestic SEO

Dixon Jones became the Marketing Director of the worlds largest link analysis engine, Majestic SEO, in 2009, transforming the SE industry by providing link intelligence on a scale not previously open to the industry. Bringing the technology to the search marketing industry has resulted in two best SEO Technology awards and a user base of thousands from WordTracker to Microsoft. The data provides users with detailed link data from anchor text to crawl date. He has worked at the forefront of search marketing since 1999, since he set up the UK based search marketing consultancy, Receptional. During that time, Dixon has been a consistent presenter on the search conference circuit all over the world and has helped clients in all sectors including national papers and international search engines themselves to understand their own search strategy. His consulting clients come from a wide variety of difficult sectors, from travel to gambling.

Why do you think there has been such innovation in the search engine space over the past few years?

Big data solutions (and in particular MapReduce, a querying technology developed by Google) has really started to allow Data to drive decision making and now also drive innovation. Finding insights that would previously have taken weeks or months of data crunching can now be completed on hours or minutes. This is very much at the route of the pace of change. Many people call Big Data just another word, but there really is a structural change that has allowed businesses to interrogate data differently. This has allowed search technologies to not only cover more data sources when a user searches, but the results returned can also dig deeper into the search results giving better insight more actionable insight for the user.

By way of a case study, Majestic was always good at being able to show you within seconds the most important links into any website or web page. With an index of over a trillion URLs, thats impressive, because it is hard to order all those records in a way that gives an easy to retrieve primary key the tool traditionally used to speed up a database lookup. Now, however, instead of starting with a URL, you can also start with a keyword on our site and we will show you all the URLs or domains that are most influenced by links with that anchor text again from a similarly large data set. That is almost entirely inverting the data and without big data principals or breaking the problem across many CPUs at the same time, it would simply be impossible to handle the Volume and Velocity at the speed required for web based applications.

What do you think the future of search will look like?

I am involved in Web search, not document search, but increasingly (at enterprise level) I think these two previously distinct areas of search will start to merge. If you are a major utility company, of course you want you search to be able to uncover documents relating to your customer (or a segment of customers), but at the same time we are understanding that people are not living in a state of stasis. People move house, move jobs, change relationships, have children. They do not then inform their utility company and should not have to. The information is out there on Facebook, in public records, on LinkedIN Search will be harnessing these data signals and driving much stronger insights based on big data patterns and predictions.

Search is getting exciting again!

Simon Bain, CEO, SearchYourCloud

Simon Bain is founder & the CEO of SearchYourCloud. Simon is the Chief Architect of SearchYourClouds software. While developing the core technology for SearchYourcloud he has provided technical consultancy services, XML training and managed software projects for a number of well-known companies including IBM, ARIBA and a range of document and content management organizations. He also created an on-line voting application which became the first of its kind to be security cleared for use on interactive digital TV for the UK government and was used to power a number of local elections held around the UK.

What is your assessment of todays enterprise search industry?

The current enterprise search industry is in flux. With vendors looking at how utilization of BYOD can enhance user experience. The uptake by users of enterprise search implementations has historically been low. This has been for a number of reasons, but mainly the complexity of the interfaces and the lack of any real relevant information being returned. This has led to files being stored within the users desktop and other off-site file stores. This flux meant that users have started to move to more relevant applications and the fragmentation of search once again.

What do you think the future of search will look like?

In the future enterprise search will become more personal. With users being able to add and delete their own search sources. True federation will come in to play, and the super index will start to take a back seat to Click-Time information access. This change will mean that users gain power to control their own results, Bringing in cloud stores, internal applications, such as CRM and Doc management, as well as pulling in external non-corporate content from web sites, such as Linked-In, Facebook and other social networks. This will then give the user a 100% view of their data and information points.

Alex Gorbansky, CEO, Docurated

Alex Gorbansky is the founder and CEO of Docurated, a next generation visual knowledge management platform which solves the information retrieval problem for leading companies like Clorox, Omnicom, Netflix, Weather Channel, and many others. While document storage is cheap, information retrieval is expensive given the digital landfill of content buried in folders and repositories across organizations. Docurated enables sales, marketing, and technology teams to surface and use the exact chart or slide they need, no matter where it is stored, without slogging through folders and files. Docurated seamlessly integrates with existing folder-based repositories unlocking years of content instantly.

What is your assessment of todays enterprise search industry?

Given the advances in consumer search, employees have grown tired of inferior enterprise search solutions. Enterprise search has had to redefine itself once again. This need has come into sharper focus as storage costs continue to tumble. Taking the recent cloud service price cuts by Amazon and Google into account, it seems cloud storage will be free sooner rather than later. The popularity of such cloud storage solutions has created a situation where data gets stockpiled across the enterprise in a variety of locations. When you consider the current rate of content production, it becomes clear that effective enterprise search is now essential. At Docurated, we have sought to address the shortcomings in enterprise search head on by developing a cloud-based platform that acts as Google for your documents. Googles ubiquity means we have grown accustomed to the availability of relevant information instantaneously, anything less in the enterprise will no longer suffice.

What do you think the future of search will look like?

Going forward, there will be less emphasis placed on organizing data, instead the focus will be on effective search. Given the masses of data at our disposal, searching for specific files or folders alone will not satisfy an increasingly millennial-dominated workforce. The facility to search within the document itself is becoming vital. The Docurated platform caters for instant access to the most relevant page or slide without even having to open the document. The age of time-consuming manual tagging and naming convention programs is nearing its end. Organizational inefficiencies such as time wasted searching for documents have no place in the rapid paced business landscape of today let alone the future. Effective enterprise search can eradicate this inefficiency. Enterprise search will become instant and intuitive, paving the way for increased productivity across the enterprise.

By Cbhan Phillipson

http://www.docurated.com/enterprise-search/enterprise-search-14-industry-experts-predict-future-search