PINQ - Interrogated datasets. Faceted Search
The faceted search built into the online store - internal search - works quickly in many respects and does not load the system.
- Built into the product
- Very fast
- Doesn't load the site
- Is the main part of the infoblocks API
- Does not require website redesign
- Automatically reindexed
The client is instantly presented with pre-prepared search results - for any combination of parameters - facet. The system calculates facets for a product in advance - all possible intersections of these properties in the filter. These ready-made search sets are then issued to clients.
Why doesn't the site load?
At the moment the result is issued to the client, no calculations occur, because the result is already ready. A facet for a new product is created immediately when it is added to the sales catalog. The search is automatically re-indexed based on new products and new properties.
Benefits for clients
Advantages of faceted search Your client finds the product very quickly, easily playing with the filter settings. The client does not wait and gets results instantly. Search speed does not depend on the number of items in the catalog.
Smart filter 2.0
Your client finds the product very quickly by consistently narrowing the search query. And at every step, he instantly receives results - a list of products upon request. He does not have to wait for his request to be processed. Because the system has calculated all possible options in advance and simply issues blanks. Gradually, the online store displays fewer and fewer products as results. These products are getting closer and closer to the buyer's request.
Interactivity and multidimensionality
Choosing a product for a buyer is like a game. The client drags parameters (price, weight), switches properties (color, size), sets additional conditions (brand, material, taste, etc.) - and the system immediately rearranges the results. In this case, there can be as many customizable parameters as you like - their number does not affect the speed of generating the result.
Convenience and friendliness
With faceted navigation, even an inexperienced user can easily select a product in the store. Manipulating search tools is very convenient. In addition, the system prompts the buyer with all the parameters by which he can choose a product. The store, as it were, demonstrates to the client the main properties of the product.
The client does not wait for the request to be processed!
For example, your client buys a camera. Initially, he indicates only 3 parameters in the smart filter: price, brand, size. Its facet includes 3 intersections, there are a lot of search results, but the client receives them immediately. The client sets the weight - he needs a lightweight gadget. The system immediately, without delay, gives him a smaller list of goods. The client sets the screen size, then specifies the required functionality. In front of him are the desired goods.
Search speed
Search speed matters Search speed affects the number of purchases madeNothing irritates a customer more than having trouble finding products on your website. The client will leave for other stores if he searches for too long. Even if your store contains big choice products and many tools for filtering them. The client may not receive results from the request.
Video tutorial: Why faceted search speeds up a smart filter significantly
Using the “facet” speeds up the search within the store by an order of magnitude. In this case, the speed does not depend on the number of elements in the directory.
Why is search so slow?
Searching for a product on a storefront may take significantly longer than normal page loading times. Especially if there are a large number of products in the store and the properties of these products. Search query in many ways it creates a lot of database calls and significantly loads the site. If there are a lot of clients and requests, the search slows down significantly.
- Smart filter component – 15 times faster!
- Catalog component – 5 times faster!
Fast even without a “facet”! The product is constantly working to speed up the catalog components themselves. The “Site Speed” service shows a significant increase in speed from version to version!
Reconstruction
Constant reconstruction of indexing and search results is carried out. The content indexing algorithm is being reworked and accelerated. The quality of presentation of search results is improved - in particular, “noise” is reduced. The developers plan to display personalized data for the current client in search results.
For Developers: API Transparency
Previous view
- Speeding up the CIBlockElement::GetList method
- Full integration with smart filter
Full integration with smart filter
Now, when making settings in the administrative part, for product properties you can not only indicate the activity - whether to participate or not in the smart filter. By passing a property to the Smart Filter, you can immediately choose how to display them. In what form should the property be shown to clients: buttons, sizes, sliders, drop-down lists, lists with color selection, etc.
Show property in Smart Filter
The smart filter now looks more beautiful. Developers can easily customize and further customize its appearance.
We took a quick look at the installation and basic syntax of PINQ, a port of LINQ to PHP. In this article, we'll look at how to use PINQ to simulate the faceted search feature in MySQL.
In this article we will not cover all aspects of faceted search. Interested people can search for suitable information on the Internet.
A typical faceted search works like this:
- The user enters a keyword, or several keywords, to search. For example, “router” to search for products in which the word “router” appears in the description, keywords, category name, tags, etc.
- The site returns a list of products that match these criteria.
- The site provides several links to customize your search terms. For example, it may allow you to specify specific router manufacturers, or set a price range, or other features.
- The user can continue to specify additional search criteria in order to obtain the data set of interest.
Faceted search is quite popular and a powerful tool that can be seen on almost any e-commerce website.
Unfortunately, faceted search is not built into MySQL. So what should we do if we still use MySQL, but want to give the user this opportunity?
With PINQ, which has a similar, powerful and simple approach, we can achieve the same behavior as if we were using other database engines.
Expanding the demo from the first partNote: all code from this part, and from the first part, can be found in the repository.
In this article, we'll expand on the demo from Part 1 with a significant improvement in the form of facet search.
Let's start with index.php and add the following lines to it:
$app->get("demo2", function () use ($app) ( global $demo; $test2 = new pinqDemo\Demo($app); return $test2->test2($app, $demo->test1 ($app)); )); $app->get("demo2/facet/(key)/(value)", function ($key, $value) use ($app) ( global $demo; $test3 = new pinqDemo\Demo($app); return $test3->test3($app, $demo->test1($app), $key, $value); ));
The first route takes us to a page to view all entries that match the search by keyword. To keep the example simple, we select all books from the book_book table. It will also display the resulting data set and a set of links to specify the search criteria.
In real applications, after clicking on such links, all facet filters will adjust to the boundary values of the resulting data set. The user will thus be able to sequentially add new search conditions, for example, first select a manufacturer, then specify a price range, etc.
But in this example we will not implement this behavior - all filters will reflect the boundary values of the original data set. This is the first limitation and the first candidate for improvement in our demo.
As you can see in the code above, the actual functions are located in another file called pinqDemo.php. Let's take a look at the corresponding code that provides the faceted search feature.
Aspect classThe first step is to create a class that represents an aspect. In general, an aspect should contain several properties:
- The data it operates on ($data)
- The key by which the grouping is performed ($key)
- Key type ($type). Can be one of the following:
- specify the full string for an exact match
- indicate part of the string (usually the initial one) to search by pattern
- indicate a range of values, for grouping by range
- if the key type is a range of values, you need to define a value step to determine the lower and upper bounds of the range; or if the type is part of a string, you must specify how many first letters will be used for grouping ($range)
Grouping is the most critical part of the aspect. All aggregated information that an aspect may be able to return depends on the grouping criteria. Typically the most used search criteria are “Full String”, “Part of String”, or “Range of Values”.
Namespace classFacet ( use Pinq\ITraversable, Pinq\Traversable; class Facet ( public $data; // Original data set public $key; // field by which to group public $type; // F: entire row; S: start strings; R: range; public $range; // only plays a role if $type != F ... public function getFacet() ( $filter = ""; if ($this->type == "F") // entire line ( ... ) elseif ($this->type == "S") // start of line ( ... ) elseif ($this->type == "R") // range of values ( $ filter = $this->data ->groupBy(function($row) ( return floor($row[$this->key] / $this->range) * $this->range; )) ->select(function (ITraversable $data) ( return ["key" => $data->last()[$this->key], "count" => $data->count()]; )); ) return $filter; ) ) )
The main function of this class is to return a filtered dataset based on the original dataset and aspect properties. From the code it is clear that for different types of accounts they use various ways grouping data. In the code above, we showed what the code might look like if we group the data by a range of values in increments specified in $range .
Setting aspects and displaying source dataPublic function test2($app, $data) ( $facet = $this->getFacet($data); return $app["twig"]->render("demo2.html.twig", array("facet" = > $facet, "data" => $data)); ) private function getFacet($originalData) ( $facet = array(); $data = \Pinq\Traversable::from($originalData); // 3 creation examples different aspect objects, and return the aspects $filter1 = new \classFacet\Facet($data, "author", "F"); $filter2 = new \classFacet\Facet($data, "title", "S", 6) ; $filter3 = new \classFacet\Facet($data, "price", "R", 10); $facet[$filter1->key] = $filter1->getFacet(); $facet[$filter2->key ] = $filter2->getFacet(); $facet[$filter3->key] = $filter3->getFacet(); return $facet; )
In the getFacet() method we do the following:
- Convert the original data into a Pinq\Traversable object for further processing
- We create three aspects. The 'author' aspect will group by the author field, and implement grouping by the entire row; aspect 'title' - by the title field with grouping by part of the line (by the first 6 characters); aspect 'price' - by the price field with grouping by range (in increments of 10)
- Finally, we extract the aspects and return them to the test2 function so that they can be output to the template for display
In most cases, filters will be displayed as a line, and will lead you to view the filtered result.
We've already created a route ("demo2/facet/(key)/(value)") to display faceted search results and filter links.
The route takes two parameters, depending on the key being filtered by and the value for that key. The test3 function that is bound to this route is shown below:
Public function test3($app, $originalData, $key, $value) ( $data = \Pinq\Traversable::from($originalData); $facet = $this->getFacet($data); $filter = null; if ($key == "author") ( $filter = $data ->where(function($row) use ($value) ( return $row["author"] == $value; )) ->orderByAscending( function($row) use ($key) ( return $row["price"]; )) ; ) elseif ($key == "price") ( ... ) else //$key==title ( .. . ) return $app["twig"]->render("demo2.html.twig", array("facet" => $facet, "data" => $filter)); )
Basically, depending on the key, we apply filtering (an anonymous function in the where statement) according to the passed value and get the following set of filtered data. We can also set the order of data filtering.
Finally, we display the raw data (along with filters) in the template. This route uses the same pattern we used in "demo2".
Search Bar
- (% for k, v in facet %)
- ((k|capitalize))
- ((vv.count))((vv.key)) (%endfor%)
- (% for vv in v %)
We need to remember that the aspects generated by our application are nested arrays. At the first level, this is an array of all aspects, and, in our case, there are three of them (for author, title, price, respectively).
Each aspect has a key-value array, so we can iterate over it using normal methods.
Notice how we build the URLs for our links. We use both the outer loop key (k) and the inner loop keys (vv.key) as parameters for the route ("demo2/facet/(key)/(value)"). The size of the arrays (vv.count) is used for display in the template.
The first image shows the original data set, and the second image is filtered by price range from $0 to $10, and sorted by author.
Great, we were able to simulate faceted search in our application!
Before concluding this article, we need to take a final look at our example and determine what can be improved and what limitations we have.
Possible improvementsIn general, this is a very basic example. We've just gone over the basic syntax and concepts and implemented them as a working example. As previously stated, we have several areas that could be improved for greater flexibility.
We need to implement “overlay” search criteria, since the current example limits us to the ability to apply search filtering only to the original data set; we cannot apply faceted search to an already filtered result. This is the biggest improvement I can imagine.
RestrictionsThe facet search implemented in this article has serious limitations (which may also apply to other facet search implementations):
We fetch data from MySQL every time
This application uses the Silex framework. Like any single entry point framework like Silex, Symfony, Laravel, its index.php (or app.php) file is called every time a route is parsed and controller functions are executed.
If you look at the code in our index.php, you will notice that the following line of code:
$demo = new pinqDemo\Demo($app);
is called every time the application page is rendered, which means the following lines of code are executed each time:
Class Demo ( private $books = ""; public function __construct($app) ( $sql = "select * from book_book order by id"; $this->books = $app["db"]->fetchAll($sql ); )
Will it be better if we don't use a framework? Well, despite the fact that developing applications without frameworks is not a good idea, I can say that we will encounter the same problems: data (and state) are not saved between different HTTP requests. This is a fundamental characteristic of HTTP. This can be avoided by using caching mechanisms.
We saved several SQL queries by using aspects. Instead of passing one select query to retrieve the data, and three group by queries with corresponding where clauses, we ran just one where query, and used PINQ to get the aggregated information.
ConclusionIn this part, we implemented the ability to facet search a collection of books. As I said, this is just a small example, which has room for improvement, and which has a number of limitations.
( "query": ( "and": [ ( "terms": ("country": ["be", "fr"]) ), ( "terms": ("category": ["books", "movies" "]) ) ] ) )
For counters, we can use built-in aggregations from Elasticsearch. Each of the two facets is stored as a single field in the index, so we can use aggregation of terms in each of these fields. The aggregation will return a counter for the value of this field.
( "query": ( "and": [ ( "terms": ("country": ["be", "fr"]) ), ( "terms": ("category": ["books", "movies" "]) ) ]), "aggregations": ( "countries": ( "terms": ("field": "country")), "categories": ( "terms": ("field": "category") ) ) )
If you were to run this query, you will notice that the counters are disabled. The two unselected countries, Portugal and Brazil, have a counter of 0. Although there are actual results if we want to select them (due to the ORinner edge). This happens because, by default, Elasticsearch performs its aggregations on the result set. This means that if you select France, the other country's filters will have a score of 0 because the result set only contains items from France.
To fix this, we need to tell Elasticsearch to perform the aggregation on the entire dataset, ignoring the query. We can do this by defining our clusters as global.
( "query": ( "and": [ ( "terms": ("country": ["be", "fr"]) ), ( "terms": ("category": ["books", "movies" "]) ) ]), "aggregations": ( "all_products": ( "global": (), "aggregations": ( "countries": ( "terms": ("field": "country")), " categories": ( "terms": ("field": "category") ) ) ) ) )
If we just did this, our counters would always be the same because they would always count on the entire data set, regardless of our filters. Our units need to get a little more complex, for this to work we need to add filters to them. Each aggregation must rely on a data set with all filters applied except its own. Thus, the aggregation by account in France counts on the data set with the category filter applied, but not the country filter:
( "query": ( "and": [ ( "terms": ("country": ["be", "fr"]) ), ( "terms": ("category": ["books", "movies" "]) ) ]), "aggregations": ( "all_products": ( "global": (), "aggregations": ( "countries": ( "filter": ( "and": [ ( "terms": ( "category": ["books","movies"]) ) ] ), "aggregations": ( "filtered_countries": ( "terms": ("field": "country") ) ) ), "categories": ( "filter": ( "and": [ ( "terms": ("country": ["be","fr"]) ) ]), "aggregations": ( "filtered_categories": ( "terms": (" field": "category") ) ) ) ) ) ) )
( "took": 153, "timed_out": false, "_shards": ( "total": 5, "successful": 5, "failed": 0), "hits": ( "total": 3, "max_score ": 0, "hits": ["..."]), "aggregations": ( "all_products": ( "doc_count": 21, "filtered categories": ( "doc_count": 13, "categories": ( "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ ( "key": "movies", "doc_count": 6 ), ( "key": "music", "doc_count": 4 ), ( "key": "books", "doc_count": 3 ) ] ) ), "filtered_countries": ( "doc_count": 15, "countries": ( "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ ( "key": "fr", "doc_count": 6 ), ( "key": "br", "doc_count": 4 ), ( "key": "be", "doc_count": 3 ), ( "key": "pt", "doc_count": 2 ) ] ) ) ) ) )
Yii2 framework $terms = QueryHelper::terms("categories.name" , "my category" ) ; $nested = QueryHelper:: nested ("string_facet" , QueryHelper:: filter ([ QueryHelper:: term ("string_facet.facet_name" , [ "value" => $id , "boost" => 1 ] ) , QueryHelper:: term ("string_facet.facet_value" , [ "value" => $value , "boost" => 1 ] ) , ] ) ) ; $filter = QueryHelper::should ($nested) ;In today's lesson we will try to recreate an imitation of faceted search using Javascript. I'm assuming you already know what faceted search is if you're reading this tutorial, otherwise google it or check out Amazon or my Demo.
First we need the library github.com/eikes/facetedsearch. Download it and connect the facetedsearch.js file to our project. We will also need the jQuery and Underscore libraries.
Disclaimer: I understand that JQ is no longer a cake, but I use it as familiar syntactic sugar, you can rewrite it for libraries more familiar to you or in vanilla JS.
So, first, let's make a simple markup with connected dependencies:
Document // Here we will display facet filters // And here our elements will be
Now we need to describe the settings of our application and create a template for displaying array elements that we will sort using facets:
$(function())( var item_template = // Describe the template "" + " 
Well, actually create a JSON array itself with elements to display in our faceted search in JS:
Var items = [ ( "firstname": "Mary", "lastname": "Smith", "imageURL": "http://lorempixel.com/150/150/cats/2", "description": "Sed Ea Amet. Stet Voluptua. Nonumy Magna Takimata ", "category": "Mouse", "language": ["Smalltalk", "XSLT"], "continent": "Africa" ), ( "firstname": "Patricia", "lastname": "Johnson", "imageURL": "http://lorempixel.com/150/150/cats/3", "description": "Ut Takimata Sit Aliquyam Labore Aliquyam Sit Sit Lorem Amet. Ipsum Rebum." , "category": "Lion", "continent": "North America", ... ];
I would put this array into a separate JS file that would be generated dynamically, from a database, for example.
That's all, we get a faceted search in JavaScript and can customize it. Next, I provide translated documentation of the library, where you can see the triggers you need.
Documentation FeaturesTwo functions are exported to the jQuery namespace.
facetelize Used to initialize a faceted search with the given settings.
facetUpdate Can be used if you want to change the facet lookup state externally.
Object settingsitems: An array of items that will be filtered and sorted in the process.
facets: An object whose keys correspond to element keys and values is the header for that facet. Items will be filtered based on what value they have for these keys.
orderByOptions: Similar to facets, except these key-value pairs are used only for sorting. When the RANDOM key is enabled, the results can be randomized.
facetSelector: This is a selector that is used to find a DOM node from which to select facet filters.
resultSelector: This is a selector that is used to find the DOM node where results are displayed.
resultTemplate: A string that is used by the Underscore template engine to render each element from the items array. The following attributes are added to each element, which can also be used in the template: batchItemNr, batchItemCount, and totalItemCount.
state: This object stores the current filters, sorts: currentResult and others. You can provide an orderBy string or a filters object to preset them.
enablePagination: Boolean to enable pagination and the "load more" button, defaults to true .
paginationCount: If paginator is enabled, sets the number of elements per page, default is 50.
facetSortOption: Use this function to change the order of facet elements. Takes an object whose keys correspond to facet names and values into an array of facet values, which can be arranged in the order you would like them to be. This example will sort the continents in a different order, adding items not included in the array in alphabetical order:
FacetSortOption: ("continent": ["North America", "South America"])
There are some more templates, please have a look source facetedsearch.js to see all available template options.
EventsYou can bind to some events which should send notifications when some actions happened. To do this, we use the jquery event system:
facetuicreated: You can bind this function to the settings.facetSelector DOM element which should be notified when the UI has been created.
facetedsearchresultupdate: You can bind this function to the settings.resultSelector DOM element to be notified of the update results.
facetedsearchfacetclick: This event is fired when a facet is clicked and fired on the settings.facetSelector element. Which receives the facet id as an argument.
facetedsearchorderby: This event is fired when the sorting element is clicked on the settings.facetSelector element. It takes the ID order as an argument.
$(settings.resultSelector).bind("facetedsearchresultupdate", function())( // do something, maybe ));
Modern people are trying to spend less and less time on shopping. Slow product catalogs drive away customers, the store loses customers and part of its profits. Make your online store more attractive with faceted technology Faceted - i.e. predefined. search. Create faceted indexes and significantly speed up the search for products and the work of the entire catalog.
Note: The faceted search mechanism is available from version 15.0.1 of the Information Blocks module and is integrated with the component. A component is a program code designed in a visual shell that performs a specific function of a module to display data in the Public part. We can insert this block of code into website pages without writing any code directly. Smart filter The component prepares a filter for selecting from an information block and displays a filter form for filtering elements. The component must be connected before the component for displaying catalog elements, otherwise the list of elements will not be filtered. The component is standard, included in the module distribution and contains three templates: .default , visual_horizontal and visual_vertical . (The last two templates are not supported, they remain to maintain compatibility.)
In the visual editor, the component is located along the path Content > Catalog > Smart filter.
The component belongs to the Information blocks module.
| Let's look at an example: We go to the online store and select in the filter that we need a red T-shirt:
|
Let's create faceted indexes in a few simple steps:
Faceted indexes are recreated automatically or you need to recreate them manually, depending on the actions performed:
|
Faceted search improves the performance of the product catalog. To use it you need:
Publications on the topic
|
