Why Open-Source (also) Matters in eCommerce
There are plenty of articles already out there that dig into this question and list the different pros and cons. But as in most cases, the honest answer is “it depends”. So, I want to keep it short and pick – from my perspective – the biggest advantage and the main disadvantage of using open source in the context of eCommerce – or more specifically when it comes to a search solution. Along the way, I’m introducing the Open Commerce Search Stack (OCSS) and show, how it leverages that advantage and reduces the disadvantage. Let’s dig in!
Pro: Don’t Reinvent the Wheel
Search is quite a complex topic. Even for bigger players, it requires a lot of time to build something new. There are already outstanding open-source solutions available. No matter if you’re eager to use some fancy AI or just a standard search solution. However, your solution won’t make a difference as long as it hasn’t solved the basic issues.
In the case of e-commerce search, these are things like data indexation, synonym handling, and faceting. Not to forget operational topics like high availability and scalability. Even companies with a strong focus on search have failed in this area. So why bother with that stuff, when you can get it for free?
Solutions like Solr and Elasticsearch offer a good basis to get started with the essentials. In this way, you can implement the nice ideas and special features that differentiate your solution. In my opinion this is what matters in the end, and where SaaS solutions come to their limit: you can only ever get as good as the SaaS service you’re using.
Con: Steep learning curve
Where we see demand for a search solution
So, why are we building the next search solution? A few years ago, we started a proof of concept to see if and how we can build a product search solution with Elasticsearch. We found a very nice guideline and implemented most of it. But even with that guideline and some years of experience, it took us quite a few months to get to a feasible solution.
The most significant difference to most SaaS solutions is the complex API of Elasticsearch. To get at least some relevant results, you have to build the correct elasticsearch-queries respective of the search query. The same applies to getting the correct facets and to implement filtering correctly and so on. It’s mostly the same case for Solr. As a result, someone unfamiliar with these topics, is going to need more time to get it right. In comparison, proprietary solutions come with impressive REST APIs that only require basic search and filter information.
We are introducing Open Commerce Search Stack into this gap: a slim layer between your platform and existing open-source solutions. It comes with a simple API for indexation and searching. This way it hides all the complexity of search. Instead of reinventing the wheel, we care about building a nice tire – so to speak – for existing wheel rims out there. At the same time, we lower the learning curve. The result is a solution to get you up and running more quickly without having to mess with all the tiny details. Of course, it also comes with all the other advantages of open source, like flexibility and extendibility, so you always have the option to dive deeper.
Our Goals for Open Commerce Search Stack
To sum it up, these are the main goals we focused on when building the OCSS:
- Extend what’s there: To this end, we take Elasticsearch off the shelf and use best practices to focus only on filling the gaps.
- Lower the learning curve: With a simple API on top of our solution we hide the complexity of building the correct queries to achieve relevant results. We also prepared a default configuration, that should fit 80% of all use-cases.
- Keep it flexible: All the crucial parts are configurable. But with batteries included: the stack already comes with a proved and tested default configuration.
- Keep it extendible: We plan to implement some minimal plugin mechanics to run custom code for indexation, query creation, and faceting.
- Open for change: With separated components and the API-first approach, we don’t bind to the usage of Elasticsearch. For example we used pure Lucene to build the Auto-Suggest functionality. So it is easy to adopt other search solutions (even proprietary ones) using that API.
Open Commerce Search Stack – Architecture Overview
We’re just at the start, so there are only basic components in place. But more are on the horizon. Already, it’s possible to fulfill the major requirements for a search solution.
- Indexer Service: Takes care of transforming standard key-value data into the correct structure, perfectly prepared for the search service. All controlled by configuration – even some data wrangling logic.
- Search Service: Hidden behind the simple Search API (you can start with “q=your+term”) a quite complex logic cares about the results. It analyzes the passed search terms and, depending on their characteristic, it uses different techniques to search the indexed data. It also contains “fallback queries” that try some query relaxation in case the first try didn’t succeed.
- Auto-Suggest: With a data-pull approach, it’s independent of Elasticsearch and still scalable. We use the same service to build our SmartSuggest module, but with cleansed and enriched searchHub data.
- Configuration Service: Since the Indexer and Search Service are built with Spring Boot, we use Spring Cloud Config to distribute the configuration to these services. However, we’re already planning to build a solution that also allows changing the configuration – of course with a nice REST API.
You are welcome to take a look at the current state. In the next installment of this series, I will present a simple “getting started”, so you can get your hands dirty – well, only as much as necessary.
