Solution Faceted Search

Search is an integral part of every company with online presence and corpus data both structured and unstructured. The goal is always to navigate customer find the product/content in a fast efficient way to purchase or view.

What is a facet: As per the Google definition, it's a particular feature or an aspect of something. For those of us from SQL background think of facets as groupBy functionality. As we all understand to perform a groupBy is a complex performance operation from code. In world of searchandize, each facet is modeled as a Dimensions and Dimension Values using a Search engine(It could be any engine Endeca or SOLR or something else)

E.g. Dimensions: Color, Size, Price, Men's, Shirts
Dimensions Values: Red,Blue,Black, White
Ranges: In case where you have e.g. Authors on books or Price where we have a lot of values, it is a good idea to go with a range.

Faceted Search process:
Most of the Search engines have an ETL process to merge\Join data from various sources and massage\clean data, if you are data is not good (e.g. If a company sells USB drive, GB would be a facet but if the value is not clean as an G B or GB. or G.B for GB, it would translate to wrong facets in the front end). hence it is important to have good data.
Once you have get past cleaning data, the process is to build Attribute dictionary and map it to the data and build indices off it.
From the front end, the Navigation is driven by state N=0 that would provide all data from Search and as you drill to specific data, you change the Navigation state. Most of the times, the navigation state is encrypted for front end customers to make the URL smaller. There are other parameters such as Navigation filters.

Endeca | Programmer primer | Objects Query\Results\Connection

A lot of customers use Endeca or SOLR as their search engine in WCS world and this primer series on Endeca is to provide some basics:

Information Transformation Layer: Transforms the source data into indices for MDEX Engine. The transformation process does not change the original data but only it's representation within your Endeca Impl.

Query (Object ENEQuery or sub class UrlENEQuery): Endeca follows one query one page approach. The MDEX is designed to return all of the information required to build an entire page.The MDEX engine is specifically designed this way to reduce the number of queries. There are four kinds of queries:

• Navigation (N) --setNavDescriptors(),
• Endeca Record (R)- setERecSpec(), 
• Dimension Search(D) --setDimSearchTerms(),
• Aggregate Endeca Record (A, An, Au)--setAggrERecSpec.

e.g. ENEQuery nequery = new UrlENEQuery(request.getQueryString(), “UTF-8”);
     ENEQuery nequery = new ENEQuery();
    DimValIdList descriptors = new DimValIdList("123");
   nequery.setNavDescriptors(descriptors);


Query Search results as EnQueryResults object: For any given type of query such as navigation, record search, dimension search, .etc. The top level object returned is EnQueryResults and that could encapsulate other objects.

Endeca Record Object as ERec Object: It contains all the information regarding the Endeca record including Endeca Properties (name\value pairs) and Information about record's location within the dimension hierarchy.

Connectivity object HttpENEConnection: enables connections with MDEX engine.
     e.g. ENEConnection nec = new HttpENEConnection(eneHost, enePort);


Query Exceptions : There are multiple exceptions ENException, ENEConnectionException objects.


Sample code for retrieval of query results:

//Create an ENEConnection
ENEConnection nec = new HttpENEConnection(eneHost, enePort);
//Create a query from the browser request query string
ENEQuery nequery = new UrlENEQuery(request.getQueryString(),
“UTF-8”);
//Execute the MDEX Engine query
ENEQueryResults results = nec.query(nequery);

Endeca overview ---High level Architecture

Endeca is a very widely used search engine for Web and E-commerce applications and it is very popular with lot of WCS clients. High level architecture components of include:

MDEX Engine:  The MDEX engine is the indexing and querying engine that provides the core functionality for all Endeca. MDEX Engine is stateless and hence requires the query to have all the parameters to be provided each time.

Endeca Workbench: The Endeca workbench is a web based tools that allows business users to configure and administer an Endeca application for Merchandising, content spotlighting, search configuration and usage reporting. The main functions are as follows
Rule Manager: Add\Edit\Activate\Deactivate rules and preview using the preview application.
KeyWord Redirects: Create and manage keyword redirects.
There are multiple functions such as Resources Lock Page, Instance Configuration Page, Dimension Order, Stop Words Page, Phrases Page, Thesaurus, Daily Reports,User Management  and EAC settings, EAC Admin Console.

Endeca Platform Services:  This consists of mainly Endeca Information Transformation Layer and Endeca application controller (EAC).
EAC consists of EAC central server and agents. EAC centeral server manages all agents.
Data Foundry: consists of forget program and related components.
Presentation and logging API's: API's to MDEX Engine and Log Server.
Logging and Reporting system.
Reference Implementation: Sample developer studio project source data for multiple platforms.
emgr_update: Utility to update instance configuration to Endeca workbench.
Endeca Control System: The Job control Daemon (JCD) and the control Interpreter.
     Forging \ETL Server: Process of gather data from various sources and building the indices is Forging.
     Pipelines: Data handling components that transforms Raw data to Indices and Deploy them


Endeca Presentation API: provides interfaces to log server and MDEX Server. The API is available in 2 flavors for Unix and Windows.

Securing webserver 404/403 requests from fingerprinting

As a part of improving the security on a eCommerce site, it is always a good idea to turn off fingerprinting.

How to reproduce this: Go to any static asset on the site and change it to wrong file name and you will see a 404 or even 403 errors. It would print all the information regarding the web server, which could be potentially exploited.

IBM_HTTP_Server/X.X.X.X-PMX4623 Apache/2.1.97 (Unix) Server at host.com Port 80

In order to protect the site for security reasons, these values need to be changed in httpd.conf to make the environment more secure. Make this change and restart webserver for the change to reflect.

Test Environment:

ServerTokens Full

Production Env:
           
ServerTokens Prod


Test Environment:

ServerSignature On

Prod Environment:

ServerSignature Off

Schema.org Microdata for eCommerce sites

A Lot of studies online talk about an increase in 20-30 percent organic traffic using schema.org.
Schema.org provides a collection of shared vocabularies webmasters can use to mark up their pages in ways that can be understood by the major search engines: Google, Microsoft, and Yahoo!

Some Relevant Tags for E commerce tags:
http://schema.org/Product

e.g.:
<div itemscope itemtype="http://schema.org/Product">
  <span itemprop="name">zinog 55 inch TV</span>
  <img src="zinog-tv-55in.jpg" alt="">
  <div itemprop="aggregateRating"
    itemscope itemtype="http://schema.org/AggregateRating">
   Rated <span itemprop="ratingValue">3</span>/5
   based on <span itemprop="reviewCount">11</span> customer reviews
  </div>

http://schema.org/Offer
e.g.
 <div itemprop="offers" itemscope itemtype="http://schema.org/Offer">
    <span itemprop="price">$600.00</span>
    <link itemprop="availability" href=" http://schema.org/OutOfStock ">Out of Stock
 </div>

http://schema.org/Review
e.g.
<div itemprop="reviews" itemscope itemtype="http://schema.org/Review">
    <span itemprop="name">Yo Camera</span> -
    by <span itemprop="author">Yo Boss</span>,
    <time itemprop="publishDate" datetime="2012-01-15">Jan 15, 2012</time>
    <div itemprop="reviewRating" itemscope itemtype="http://schema.org/Rating">
      <meta itemprop="worstRating" content="1">
      <span itemprop="ratingValue">3</span>/
      <span itemprop="bestRating">5</span>stars
    </div>
    <span itemprop="description">The Yo Camera is a revolution.</span>
  </div>


This is interesting for SEO, just thought of adding it here, How Google perceives web pages.
http://www.google.com/webmasters/tools/richsnippets

Ref:
http://schema.org/

Exception| findSEOPageDefintionByPageNameAndStoreID

If you have just created your V7 instance and running into this problem, the problem could be probably hitting the wrong storeId from your browser. The tendency is to look at Stores\WEB-INF\Conf and check the new configuration files for SEO and also blame it on Feature Pack upgrade :) at least I thought so.
 I ran into this problem when I was trying to hit the URL below, the store ID is valid but it was for EliteStorefrontAssetStore
http://localhost/webapp/wcs/stores/servlet/TopCategories_10701_10001


 /GenericJSPPageError.jsp - No expression builder has been configured with the name: "findSEOPageDefintionByPageNameAndStoreID".
                                 javax.servlet.jsp.JspException: No expression builder has been configured with the name: "findSEOPageDefintionByPageNameAndStoreID".
at com.ibm.commerce.foundation.internal.client.taglib.GetDataHandler.execute(GetDataHandler.java:722)
at com.ibm.commerce.foundation.internal.client.taglib.GetDataTag.doTag(GetDataTag.java:388)
at com.ibm._jsp._TopCategoriesDisplay._jspService(_TopCategoriesDisplay.java:813)

Correct URLs for both stores for out of the box are below:

Elite-Login

Madisons-Sign In

Holiday Readiness | Performance tips for WCS sites?

1. One of the important thing to start with, DB clean jobs for guest users\guest orders\CTXMGMT\CTXDATA, need to setup and monitored regularly.
2. Expired promotions should be monitored and deactivated.--This was a problem in previous versions
3. The number of times OrderCalculateCmd is called should be minimal. This is a very heavy command, as it calls calculation engine for tax\shipping\promotions. One of the sites, I worked on this was called multiple times during merge cart, cart, checkout. It is a very heavy command. If the number of items in the cart remains same and promotions are not added at the run time, should try to avoid a call to this.
4. Junk cart should be assigned to a custom user.This is to safeguard against any valid scenario invoking JunkCart on a live site.
5. Dyna cache enhances greatly the performance and should be used as it reduces the CPU usage greatly. One could also create CacacheableControllerCommands to taking Dyna to the next level and I have used this and it helps improve performance a lot.
6. Monitor production errors and calls made on web server access logs and fixing any errors, would help enhance performance. Reducing errors helps reduce load on application server.
7. Pricing is one of the heavy calls in WCS and we have built a pricing registry to enhance the pricing performance problem, it helped greatly.
8. On a couple of sites, I worked on Ajax calls were used heavily and most of our AJAX calls are Dyna cached and that helps heavily.Also sections of the page such as cart page where calls to bring up accessories or other merchandising associations should be converted to Ajax calls and it wouldn't block the main functionality of the page such as cart.
9. DB performance, making sure the stats are gathered correctly and finding out the top 10 performance intensive queries and run maximum number of times and making sure for all these queries, the indexes are correctly defined and tune the queries to make sure they efficient (for oracle use Explain plan).
10. Make sure all custom tables have opt counter triggers other wise this could cause race conditions.
11. A lot of performance is achieved from Edge Caching. We use Akamai where all the product and category pages are completely caches. Edge servers are refreshed every night after stageprop. (This is not required if new content is not refreshed every night).
12. Number of items in cart limit, there are some bots that try to add 100's of products to cart and that it self doesn't break but a combination of that and if there are automatic promotions could cause instability. I would advise a configurable parameter for cart limit and writing custom code for this. If you do not want to do a code fix, also you can do

ALTER TABLE SCHEMA_NAME.ORDERITEMS ADD
CONSTRAINT check_orderitems_1
CHECK (quantity <= 100) ENABLE
VALIDATE

The constraint will bring the user back to home page on trying to add to cart.

13. Extra objects from memory can be controlled specially while using beans in JSP's by making sure the scope is correctly defined
e.g. Specially this starts to matter for Order tunnel pages as there are multiple imports to different JSP's. It helps by giving the scope to request.

14. Reduce the guest user creation in the system by making sure, the isGeneric returns true, if a command needs to be used for generic users. We found multiple commands when turining the isGeneric to true reduce the creation of thousands of guest users. By default isGeneric is false.

15. If you are not using OOB payments, there are some commands that still could call and populate these tables on top of disabling from wc-server.xml , the following update is required.

INSERT INTO CMDREG(STOREENT_ID, INTERFACENAME, CLASSNAME, TARGET) VALUES (0,'com.ibm.commerce.edp.commands.PrimePaymentCmd','com.ibm.commerce.edp.commands.PrimePaymentVoidCmdImpl','Local'); INSERT INTO CMDREG(STOREENT_ID, INTERFACENAME, CLASSNAME, TARGET) VALUES (0,'com.ibm.commerce.edp.commands.ReservePaymentCmd','com.ibm.commerce.edp.commands.ReservePaymentVoidCmdImpl','Local'); INSERT INTO CMDREG(STOREENT_ID, INTERFACENAME, CLASSNAME, TARGET) VALUES (0,'com.ibm.commerce.edp.commands.FinalizePaymentCmd','com.ibm.commerce.edp.commands.FinalizePaymentVoidCmdImpl','Local'); INSERT INTO CMDREG(STOREENT_ID, INTERFACENAME, CLASSNAME, TARGET) VALUES (0,'com.ibm.commerce.edp.commands.TriggerPaymentActionsCmd','com.ibm.commerce.edp.commands.TriggerPaymentActionsVoidCmdImpl','Local'); INSERT INTO CMDREG(STOREENT_ID, INTERFACENAME, CLASSNAME, TARGET) VALUES (0,'com.ibm.commerce.edp.commands.CancelOrderCmd','com.ibm.commerce.edp.commands.CancelOrderVoidCmdImpl','Local'); INSERT INTO CMDREG(STOREENT_ID, INTERFACENAME, CLASSNAME, TARGET) VALUES (0,'com.ibm.commerce.edp.commands.StoreAndValidatePaymentCmd','com.ibm.commerce.edp.commands.StoreAndValidatePaymentVoidCmdImpl','Local'); INSERT INTO CMDREG(STOREENT_ID, INTERFACENAME, CLASSNAME, TARGET) VALUES (0,'com.ibm.commerce.edp.commands.PIAddCmd','com.ibm.commerce.edp.commands.PIAddVoidCmdImpl','Local');

16. Multiple Invalid cookie errors were caused due to contention as we have multiple AJAX calls in pages and due to contention. Implemented a CustomFilter and convert requests into stateless so that commerce

can process as generic user requests.

// Check for stateless URL
if (uri != null && urlsToFilter != null && urlsToFilter.indexOf(uri) >= 0) {
request = new CustomHttpServletRequestWrapper(request, response);
response = new CustomHttpServletResponseWrapper(response);
}

chain.doFilter(request, response);