Drive B2B e-commerce modernization with search

About a year ago, my team started to engage to with a large multinational consumer goods conglomerate to help improve its B2B e-commerce customer experience platform. In this blog, I’ll discuss the challenges we tackled and the modern e-commerce search platform we helped build to enhance the end-user experience.

Legacy e-commerce system’s challenges and limitations

The platform’s e-commerce implementation at the time was based on Endeca, with the majority of the data sourced from the company’s ERP (SAP Hybris) and their Product Information Management (PIM) system. The client was faced with various challenges, including:

• Slow performance as a large part of the queries took seconds to run
• Poor relevance for search results both in type-ahead and searches
• Very limited usage of product attributes for filtering/dynamic navigators
• Significant latencies (sometimes measured in days) for applying master file changes into the searchable product catalog
• A large monolith ingestion application based on a legacy technology (hyEnd – a Hybris extension for loading data in Endeca)
• Inflexible, hard to change and poorly supported by the search engine vendor

The B2B e-commerce platform was only one component of a multi-channel mix through which the company sells products to distributors and other business customers. This mix also includes mail orders, calls to sells reps, partner channels, Electronic Data Interchange (EDI), or product sales through other legacy ERP/commerce systems that hadn't transitioned to the company’s central ERP.

The increasing importance of digital channels in a B2B omnichannel strategy

Over the last few years, significant shifts have emerged that put digital experiences at the forefront of B2B omnichannel commerce strategies for any large corporation. Findability and search are core components of many modern digital distribution platforms. A variety of research studies and industry papers highlight these changes.

<<< Start >>>

<<< End >>>

<<< Start >>>

<<< End >>>

From conversations with the client, I learned that the company was well aware of these trends and the need to modernize and expand their e-commerce platform – a decision which viewed through the prism of recent developments related to COVID-19 impacts proved to be even more timely and wise. My team helped the client implement a new e-commerce search solution which allowed them to completely migrate from Endeca, deploy a much faster, relevant and easy-to-use search experience, and reap the benefits of the new search platform in a few (more than initially expected) other parts of their organization.

So what did the solution entail? I’ll describe it in detail below.

ERP systems (and their e-commerce extensions): Suboptimal for modern search experiences

Many large ERP platforms (SAP, Oracle NetSuite, Microsoft Dynamics) provide extensions for e-commerce systems but may fall short of providing the necessary capabilities to power a great search experience:

• Most of these platforms and extensions are built on relational database technologies which are great for managing master file data but aren't optimized for search.
• They don't offer federated search across multiple datasets: product catalog; product or marketing web content; products documentation: flyers, brochures, regulatory docs, PDF, etc.
• Performance is poor when querying for product catalogs with a large number of SKUs (from 500,000 to over 1 million).
• It's difficult or impossible to search and filter on fields for very wide or sparse data sets (product catalogs with thousands of attributes).
• Support for full text search is poor (product descriptions, names, etc.). The situation becomes even more complicated when support for multiple languages is required.

For our client's use case, we chose to use the best tool for the job: deploy a search engine that joins data from the company’s ERP and various systems into a unified experience that integrates seamlessly with the company’s B2B store.

Search-based solutions, skillfully designed and implemented, are the answer

The new system is based on Elasticsearch and provides a number of key features:

#1.   Great support for both structured and unstructured content

• Structured: product attributes, price, product categories, etc.
• Unstructured: product descriptions, product names, product features, etc.
• Documents: flyers, brochures, regulatory and safety data sheets, etc.

#2.   Fast, scalable, overall lower hardware infrastructure costs

• Horizontal scalability: parallel queries over large datasets
  • In our example, there are about 2 million SKUs in the system (over 60 countries)
• Faster searches: searches are now measured in hundreds of milliseconds versus seconds
  • Especially important for type-ahead (previously too slow to be used)
  • Now enhanced with suggestions and product images
• Faster ingestion: daily full refresh reduced from 20 to just 7 hours
  • Plus incremental updates which keep the site fresh throughout the day
• Lower hardware costs: larger ingestion and search workloads supported with less than half the hardware the old solution utilized

#3.   Improved search relevance

• Powerful relevance models: customizable in a variety of ways
  
  • Special parsers that account for various languages nuances (over 40 languages are supported)
  • Elasticsearch comes with support for most popular languages out-of-the box and community plugins are available for many other languages
• Support for type-ahead/suggestions: product names, brands, categories, and identifiers
• Product identifiers search: represent a large part of use cases
  • Extensive support for searches on subparts of the product identifier, where typical keyword-based search approaches don't work
  • E.g. Universal Product Codes (UPC) (various variants), manufacturer and customer specific part designations

#4.   Complex filtering and aggregations

• Deep and wide product catalog: difficult to filter and aggregate with relational database technologies
  • Thousands of product categories in a hierarchy that goes up to 6 levels deep
  • Over 10,000 attributes across the entire catalog
  • Many products have anywhere from 10 to over 100 attributes, sometimes inherited from parent categories in the hierarchy
  • Attributes can have their own properties: how and where they should be displayed, data types (numeric, enumerations, boolean, text), units of measure, etc.
  • Attributes can also have values localized for all the countries were products are sold
• Truly dynamic filters and navigators: most relevant attributes to be provided as filters determined at query time based on (the old system used to have a hardcoded list)
  • The product category selected; or 
  • The totality of search results for a query
• Complex authorizations rules: each user (customer or distributor) of the B2B store is given permissions to buy, sample, or view only a specific set of products
  • Permissions granted through a complex and hierarchical security model
  • Sophisticated filters to ensure all authorizations are properly honored

#5.   Rich content ingestion

• Deployment of Accenture’s Aspire connectors: wide support a variety of content sources
  • Allows for the ingestion of rich content (web crawls, parsing of MS Office, PDF documents)
  • Enhanced support for federated searches: both product catalogs and media library can be searched at the same time

#6.   Self-learning capabilities

• Search and type-ahead activity logs and call to action (CTA) interactions: every query, filter, pagination, add to cart, view product page, compare products, share products, etc. are fed back into the search engine and used:
  • For measuring the quality of the search results using search engine scoring techniques
  • As input datasets for enhancing the search experience (e.g. providing typeahead suggestions based on past query history)
  • As signals (together with sales data) to improve the search relevance models

A reference architecture of the new search platform is depicted here:

<<< Start >>>

B2B/C E-commerce search platform unifying content from various enterprise repositories

<<< End >>>

• The ingestion pipelines have been developed as Java Spring Batch and Spark applications running as pods in Kubernetes. Kafka was used for decoupling dependencies between components and exposing data produced by these pipelines to other systems.
• The big hyEnd monolith has been replaced with many smaller and modular components that are much easier to maintain and understand by the developers who support the system.
• All components (over 60 of them) share a common set of services/libraries for interactions with external systems and reusable plumbing functionality. They're all coordinated and orchestrated through Airflow – which incidentally also provides an excellent way to document and visualize the complex business workflows.
• The API servers are deployed as microservices in Kubernetes. The reference Kubernetes platform in this use case is OpenShift – we make heavy use of OpenShift Deployment Configs and blue/green deployment strategies for both ingestion and APIs. This allows us to perform all of our deployments with zero downtime for the site.

Increase ROI with various uses for the new search platform

At the start of the project, the push of the new investment was to deploy a search platform underpinning the search experience for both:

1. the company’s B2B Store; and
2. the B2C/consumer-facing product store and catalog.

The system has been architected in a way that supports both use cases. The first use case has been deployed to production; and the product catalog ingested into Elasticsearch already includes the entire set of products needed for the B2C site. We've started building APIs for the B2C store – more to follow on this on a later B2C focused blog post.

As the platform grew, new use cases have popped up and multiple internal teams within the company have started to replace their product extract and search capabilities with APIs exposed through the new search platform and drove requests for expansion, such as:

• Large/bulk data extracts for pricing and product reports
• Internal portals (Customer Support and Sales Rep applications)
  • Including a Salesforce application which offered a consistent view and experience for product catalog data for both B2B customers and sales reps.
• Product syndication with external channel partners
• Better alternative for some query use cases that were accomplished through the relational enterprise data warehouse (EDW)
  • For many use cases, business analysts can leverage faster, more powerful and interactive dashboards through Kibana versus the BI tool deployed with the EDW.
• Extensions for adding of Orders/Invoice History in the data sets that can be queried through the federated search
• Product recommendations based on past order history
• Expose the Search APIs for headless e-commerce: machine-to-machine interactions with technology savvy distributors and partners

These unexpected benefits furthered improved the ROI.

It’s important to note that we worked closely with many employees and consultants on the client's side. This strong partnership was built on a mutual learning experience: the client's architects and developers have been essential in helping us understand their business and internal systems. And by working side-by-side, our team helped the client's team learn new technologies and techniques so they can confidently maintain and enhance their search platform in the long-run.

Know your users to optimize their search experiences

The new external competitive pressures and challenging economic conditions have driven enterprises to modernize their digital presence to remain relevant and grow. B2B e-commerce is a key component of this strategy. And providing customers, external partners, and internal stakeholders with a unified, well curated, easy, and fast search across relevant data sets, is a core driver for e-commerce success. We’ve found that enhancing the user’s digital experience is the optimal path for many e-commerce clients. As a Forrester paper has shown, using self-serve website, price optimization, and product personalization enabled by website and search data can lower customer acquisition costs and increase customer lifetime revenue.

But where do we start? Knowing the people who are executing the searches is crucial.

Because this is B2B, your end-users are business people filling orders and invoices, often working for purchasing departments. These aren't relevancy-ranked search results like you would find on Google.com. Instead, your end-users will typically be looking for something very specific, using identifiers, dimensions, product names, etc. so they can add the right items to their orders. So, as always, “know your users” and design the search system to exactly fill their needs and methods for doing business.

It’s more important now than ever to bring the best possible digital experience to your customers, employees, and stakeholders. How can your organization deliver more value with faster, more intelligent e-commerce search? Connect with us to learn more and see how we can help.

---

Deliver more value to your users with intelligent e-commerce search

connect with us

---

<<< End >>>