Sunday, November 21, 2010

Cacheable Command ! a really cool performance addition

If you are using Dynacache for you commerce site. This functionality is one of the coolest features that could be incorporated to improve caching results drastically.
It has 2 parts:
1. Writing the actual cache able command extending CacheableCommandImpl
2. Making the entry in cachespec.xml

In the command implementation, mandatory methods to be implemented
public void setOutputProperties(TargetableCommand fromCommand)
public void performExecute() throws Exception
public boolean isReadyToCallExecute() {

Good candidates for this kind of caching would be any complex computational logic such as Geo lookup, ATP messages or any complex logic that requires lot of computation and if the underlying data changes dynamically, will need to also build the cache invalidation with it.

STAGEPROP ! Mystery behind the starting number !

I have worked on stageprop over the years on several issues but this is an interesting one that I had to wake up really early morning to help resolve.
Lots of clients have millions of records to STAGEPROP so some clients use a batch commits to avoid memory problems from loading the whole set. The usage of STGPKEY is very important to remember, if there is a STAGEPROP issue where the STAGELOG processed flag is manually changed states from unprocessed to processed and back. It is very important to understand that the starting number in STAGEPROP comes from the query below.

select stgpkey from staglog where stgrfnbr=0;

Thursday, November 18, 2010

Show after discount prices in Product page\Category pages

If you have 6.0.0.9 or higher versions of commerce. You can use the bean below to retrieve price after discounts.

PromotionalPriceDataBean

Before this bean, it was very performance intensive and complex to retrieve price after discount prices.

It is important to understand that it works by creating a temp order and out of the box, it only works by sending 1 catentry at a time, for multiple items, this bean needs to be extended.

The price is calculated as if an order is placed that includes only the current catalog entry with quantity of one. I haven't used this in a live site hence I am curious about the performance of this bean.

Sunday, November 7, 2010

Debugging, memory problem at the appserver

In the past when we saw there were memory issues that brought down the entire site. We have used a few methods but this was the most effective and helped us figure out the issue. Heap dumps are pretty good but they take a long time and tools have size/JDK/OS limitations to load them.

Application servers > server1 > Process Definition > Java Virtual Machine

Set this parameter in the Generic JVM arguments
-XX:+PrintClassHistogram

The histogram gave a pretty good idea of the objects in memory that helped us trace the issue.

Wednesday, November 3, 2010

Pillars of SEO !

1. To have short URLs with productnames, categorynames and having keywords in these links.
2. Canonical URL links in the HTML header.
This is for the search engines to look at this tag and to digest the page in question should be treated as though it were a copy of the URL and that all of the link & content metrics the engines apply should technically flow back to that URL.
e.g.

<link rel="canonical" href="http://hostname/product/en/abc350" />
<link rel="canonical" href="http://hostname/category/en/s_furniturechair" />

3. Generating Google site map periodically.
4. XML site map and keeping at the root of the shop.
5. Link building and using Fat Navigation\Fat Footer With using key words in links.

Monday, November 1, 2010

Common Errors that could bring down a site?

1. If you have a couple of traces on Websphere or Commerce in production environment that is good enough to bring down the eCommerce site.
2. TrashCan user disabled. This could really spike up the CPU utilization and could also cause DB locks.
3. Dynacache servlet caching disabled, the CPU on the application and DB server would spike up that could potentially bring down the server.
4. Cart with number of items greater than 50 added to an individual cart with active promotions against those items and if the quantity is increased to a big number.
5. Not correctly setting this parameter could bring down the server.
Application servers -> web container transport chains ->WCInboundDefault->TCP Inbound Channel (TCP_2) -> Maximum open connections. (Specifies the maximum number of connections that are available for a server to use.)

Tuesday, October 26, 2010

APAR Installation| How to do it efficiently and reduce downtime

Generally lot of clients bring down all servers as a part of the APAR\Code installation.

There is an alternate approach that would help a lot with down time.

As most of our APAR’s could be only commerce code changes this would work perfectly, if there are DB changes this approach may or may not work and the APAR might need analysis to see the nature of the DB changes.

Steps:

1. Bring down the primary node. (Each cluster has only 1 primary node and the rest are federated nodes.)
2. The site is technically still up and should be able to handle the requests.
3. Install APAR using the DM on the primary node. (All the APARs are technically always installed only primary node.)
4. Verify for successfully installation of the APAR.
5. At this point bring down other nodes.
6. Synchronize nodes and restart.

It takes about 15 minutes to do the step 5 and Step 6 and that should be the down time.

Also the same approach is used for code deployment and apparently this helps them to confirm for the code deployment and if there are issues with any code deployment, the site can still be up while the issue is worked upon.

More Information on the topology look up in info center:

IBM has APARs, Fix Packs and Feature packs for Commerce and Fix Packs and iFixes for Application Server.

Topology Background high level: IBM 6.0 Deployment: Cell\Nodes\Node Agent\DM (Network Deployment Manager)

Sunday, October 17, 2010

Why use Ajax? Using Ajax for performance reasons

AJAX is a very powerful tool to the web world and adds another dimension to the eCommerce pages.

I have used this mainly:

1. For interactive or dynamic interfaces on the web.

2. To bypass the akamai cached pages for dynamic content. If you are not using partial caching for akamai pages. It is a very powerful tool for pricing\ATP Messaging in product and category pages.

3. For performance improvements. e.g. Accessories in a cart. As these events can be asynchronous and does not have to wait for the whole page to load up.

Out of the box DOJO is provided and I have used RICO for several years, when we did the comparisons at that time, it was strictly for performance reasons that RICO was chosen. And JQUERY seems to be the preferred choice now a days.

I have used primarily for?

1. Replace the complete DIV\SPAN as AJAX response.

2. Use JSON in the Ajax response to replace selected HTML elements.

Things to do for an Ajax call to work.

1. Register the Ajax event.

2. Write the custom controller command for the Ajax and register command and response in struts-config-ext.xml

3. Ajax response JSP to replace the HTML entity.

I have found it particularly interesting to use AJAX for performance for min-carts\Accessories and other places where the complete page reload can be avoided.

Saturday, October 16, 2010

OWASP TOP 10, How WCS OOB addresses these vulnerabilities or provides frameworks to help.

The goal of this blog is to explain how the OWASP top 10 are protected in WCS out of the box.

OWASP (Open Web Application Security Project)is used to educate web application architects\developers\testers\managers regarding the most common security vulnerabilities.

Always abide these common security principles:
Accept known good.
Default Deny.
Principle of least privileges.
Using Layered Security or Defense in Depth.

The OWASP Top 10 is a list updated by OWASP every year of the top 10 security risks.

A1 - Cross Site Scripting (XSS)
A2 - Injection Flaws
A3 - Malicious File Execution
A4 - Insecure Direct Object Reference
A5 - Cross Site Request Forgery (CSRF)
A6 - Information Leakage and Improper Error Handling
A7 - Broken Authentication and Session Management
A8 - Insecure Cryptographic Storage
A9 - Insecure Communications
A10 - Failure to Restrict URL Frequently,

A1: XSS is caused by vulnerabilities such as introduce worms,hijack sesssions, etc in the sites that allow user supplied data that is not encoded and validated properly.

WCS Protection:
In wc-server.xml starting 6.0.0.4 the XSS could be specified at every module
and prohibited characters can be specified in the list.
It is a good to have a whilte list for input validation and there is a black list in wc-server.xml for restricting.
e.g.
<XSiteScriptingProtection
display="false"
enabled="true" name="Cross Site Scripting Protection">
<ProhibitedAttrs display="false"/>
<ProhibitedChars display="false">
<Character
display="false" name="&lt;SCRIPT"/>
<Character
display="false" name="&lt;%"/>
<Character
display="false" name="&amp;lt;%"/>
<Character
display="false" name="SCRIPT>"/>
<Character
display="false" name="&amp;lt;SCRIPT"/>
<Character
display="false" name="JAVASCRIPT:"/>
<Character
display="false" name="&amp;#x73;&amp;#x63;&amp;#x72;&amp;#x69;&amp;#x70;&amp;#x74;"/>
</ProhibitedChars>

it provides the input validation framework that can be used to validate the input text by regular expressions.

In JSP you can use the UIUtil to validate:
<%@page import="com.ibm.commerce.tools.util.UIUtil"%>
<%=UIUtil.toJavaScript(java.lang.String.valueOf(pageContext.getAttribute("storeId")))%>

It is always recommended to validate data on the server side, client side validation can be broken by hackers.


A2 - Injection Flaws could result in is caused modifying, deleting or viewing unauthorized data.when user specified data is passed to SQL interpreter as a command.

WCS Protection:In WCS and J2EE in general using prepared statements with specific parameters and validating the parameters is a generally used practice.
Dynamic query front end should be minimized for customer facing applications.

A3 - Malicious File Execution could result in server compromise and virus attacks and this is caused by Remote file inclusion and remote code execution that allows applications to accept files that could result in include hostile files and data.

WCS Protection: WCS being a J2EE App is executed in a JVM run by a sandbox protected by a security manager. It is very important to configure the security manger and the app is demanding permissions appropriately.
Strong user input validation helps even this vulnerability.
Firewalls for web servers should be protected and have a exact or a whitelist of ports and IP addresses that it can allow connections from.
At the OS level, a good idea to segment the file system in production and having an appropriate demarcation.


A4 - Insecure Direct Object Reference could cause unauthorized access to data and files hosted in a web application and this is caused by having direct references such as file, directory, a database key such as orderId, userId exposed in the web application html code or cookies.

WCS Protection:: It has Access controls in place for protection when exposing database keys OrderId,UserId..etc. It has a cookie WC_AUTHENTICATION_148499839 and the value is appended with a uniquely generated code.
148499839%2csFggXZ6m06QiSRLRI9c8DuYlQPc%3d
Access controls for commands and views are protected in commerce by a ACPolicy.xml and all custom commands\views\resources should be correctly defined in these files.

A5 - Cross Site Request Forgery (CSRF): It could compromise the authorized users data and using his credentials exploit other system vulnerabilities as an authorized user. This is mainly caused by remotely taking control of user's session and forging requests from the victim's browser.
XSRF could use the XSS vulnerability to exploit other system vulnerabilities.
Each request a valid unique token that is passed back and forth.

WCS Protection: WCS offers out of the box protection against this vulnerability starting with WCS FixPack 6.0.0.8.
Please find below the configuration and code required to address. The action needs to be updated in struts-config-ext.xml with csrfProtected attribute.

<action parameter="com.ibm.commerce.usermanagement.commands.UserRegistrationUpdateCmd" path="/UserRegistrationUpdate" type="com.ibm.commerce.struts.BaseAction"><set-property property="https" value="0:1"><set-property property="authenticate" value="0:0">
<set-property property="csrfProtected" value="10101:1">
</set-property>
# Edit the JSP file that invokes this action to include the authToken URL parameter.
For example:
<input name="authToken" value="${authToken}" id="WC_UserRegistrationUpdateForm_FormInput_authToken_In_Register_1" type="hidden">

Do use HTTP Post when sending sensitive data.
WCS also has configurable transaction LoginTimeout in wc-server.xml.

A6 - Information Leakage and Improper Error Handling: It could result in powerful attacks and the web application details such as configuration information, internal workings, etc can be leaked via html outputs or long error messages

WCS Protection: Out of the box provides ECApplicationException and ECSystemException and it allows all the errors generated to be forwarded to a common view.
The detailed messages can be disabled and commerce provides a framework to define user friendly messages for these exception using ECMessage and ECMessageHelper.
Making sure all the coders follow the common exception flow handling.
Also if you are using custom logging framework make sure all the sensitive data is masked in the logs.
This framework and exceptions can be extended to incorporate custom features. Some websites also tend to send the user back to home page on any system exception.


A7 - Broken Authentication and Session Management, this vulnerability could lead to breaking into user\admin passwords and also violating privacy laws. This is caused by flaws in authentication mechanisms and poor session management.

WCS Protection: It uses one way hash for the password protection and it protects the passwords from being decrypted.
WCS Commerce admin console provides an interface to define password policy and account policy and this policy is configurable and can be modified on a periodical basis.
WCS allows configuration using struts configuration to define URL's that require authentication.
WCS uses secure authentication cookie (WC_AUTHENTICATION_ID) to manage authentication data
All administrative functions are hosted on applications such as Accelerator\Admin Console\Org admin console and these are open on ports 8000,8002,8004. These ports should not be open to internet and all these users who have access to these internal applications should be governed by good password and account policies just like external users.


A8 - Insecure Cryptographic Storage
This could lead to compliance violations such as PCI and also sensitive data such as credit card information could be leaked. This is mostly caused by weak cryptographic algorithms and the key management.


WCS Protection:
WCS uses strong triple DES algorithm for encryption. That meets the PCI DSS standards.
WCS also provides KLF framework to change the encryption key periodically and allows to re encrypt the sensitive data using the new key.
Do not create new cryptographic algorithms use standard API e.g. bouncycastle.
Store private keys with extreme care.


WCS out of the box introduces keyword krypto on protected sensitive information.

e.g. Out of the box in wc-server.xml
<ProtectedParameters name="Protected Parameters">
<Parameter display="false" name="cardNumber"/>
<Parameter display="false" name="password"/>
<Parameter display="false" name="passwordVerify"/>
<Parameter display="false" name="passwordOld"/>
</ProtectedParameters>

Also there are 2 other elements that should be defined appropriately ProtectedMultiValuedParameters and NonEncryptedParameters

A9 - Insecure Communications
This would result in compliance failures and could expose the private data for spoofing.
This is resulted from not using encrypted network traffic. Opening your webservers to a limited blacklists of Port and IP Addresses.

WCS Protection:
Using SSL when transmitting sensitive data can be achieved by configuring the corresponding actions and views
e.g.
<set-property property="authenticate" value="0:1"/>
<set-property property="https" value="0:1"/>

For back end applications communicating with external applications, make sure the firewall is opened for specific IP Addresses any communication required.

A10 - Failure to Restrict URL Frequently, could result in exposing a certain function or data to unauthorized users. This could be resulted from not having access control checks to protect resources\urls

WCS Protection: out of the box offers access control mechanism and also roles for segmenting authorized content. The access control matrix should be included as a apart of the design and development and URL's and Actions are appropriately protected.

The access control framework: It divides Views and Commands that require authentication and authorization is handled by the access control policies.


This was the list from earlier to 2010.

2010 has a couple more vulnerabilities
A10:

Unvalidated Redirects and Forwards

Add a URLRedirectFilter element in the Module element as shown in the following example:
<module contextpath="/webapp/wcs/stores" fileservletenabled="false" name="Stores" urlmappingpath="/servlet" webalias="/wcsstore">
<initparameters adapters="XML/HTTP, BrowserAdapter" contextsetname="Store" handledoubleclick="true">
<urlredirectfilter enable="true">
<allowedhost name="www.mycompany1.com">
<allowedhost name="www.mycompany2.com">
<alloweddomain name="mycompany3.com">
</alloweddomain>
</allowedhost>

A6: Security Misconfiguration :

Applying the latest patches at OS\DB\Application server and Commerce server and making sure the security configuration is correctly done.



ref:
http://www.owasp.org/index.php/Main_Page

Tuesday, October 5, 2010

DB Clean | performance of overall site

DB Clean:
In my opinion next to Dynacache\EdgeCaching strategies for a site. I would consider DB Clean a very important aspect of improving site performance for WCS. It would also prevent any junk car issues.

Out of the box, commerce provides a table for adding these queries CLEANCONF table, which is good enough for most basic queries for complex cleanup operations, it is very important to use store procedures.

High Level commerce deletes required for
1. Address
2. CtxMgmt
3. Guest users
4. Orders
5. Schedulers
6. Staglog
7. CustomData

e.g.
insert into cleanconf (OBJECTNAME, TYPE, STATEMENT, NAMEARG, SEQUENCE, DAYSARG)
values (
'activity', 'remove-obsolete-ctxmgmt',
'delete from ctxmgmt where lastaccesstime < (sysdate - ?)',
'no', 1, 'yes');

Crontab utility in unix/solaris can be used to schedule these jobs. Usually it is very important to wrap this in a shell script and use some logging and also capture the PID, if you need to kill a process.

Tuesday, August 10, 2010

Multiple Sessions/LogonId in Commerce 6.0 ? not supported OOB

Authentication framework in Commerce:
The current WCS 6.0 implementation supports a single session at any given time for a user and when you try to login from a different browser/Device using the same loginId. It kills the other active session.

Details:
Essentially the context of a user in WCS is handled by a serviced called Business Context Services, essentially it folds into the Business context engine which controls the behavior of the business components over the lifetime of a session.
It uses a couple of tables CTXDATA and CTXMGMT, to manage the context. Essentially each time a user comes to the site, a new activity is created in CTXMGMT and
it associates the calling user (Users_id as CALLER_ID) with an ACTIVITY_ID . The ACTIVITY_ID remains the same in the course of the session and it also keeps all the contexts associated with the activity in CTXDATA table.

When the same user logs in from a different device or a separate browser, essentially the previous context in CTXMGMT is marked as T (Terminated). It creates a new activity with a state A (Active).

Out of the box this is the behavior and I don’t know if commerce provides any API to customize multiple activities active. I know from my past experience, whenever a user has multiple activities active due to any discrepancies, he cannot login into the site at all and gets a business context exception. I can think of a customization to manage activities outside of business context and mapping them to the same user.

http://publib.boulder.ibm.com/infocenter/wchelp/v6r0m0/index.jsp?topic=/com.ibm.commerce.developer.doc/concepts/csdbcsbcm.htm

Monday, August 9, 2010

Learning from a DBA

I have a good friend who was visiting for a conference and we had a chance to catchup on things and he is a total enthusiast for technology and database and was kind enough to go over some tips for tuning DB for development environments.

1) Problem statement, Oracle 10g instance on windows takes by default around 600MB RAM? Reduce to 300MB

Login as:
sqlplus \ as sysdba

SQL>show parameter sga;

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
lock_sga boolean FALSE
pre_page_sga boolean FALSE
sga_max_size big integer 584M
sga_target big integer 584M


SQL>alter system set sga_target = 300M scope=both;

SQL>alter system set sga_max_size = 300M scope=spfile;

Dynamic changes in Oracle are defined by giving the parameter scope=both and changes that need a restart are defined by scope=spfile;

SQL>shutdown immediate
SQL>startup


show parameter sga;

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
lock_sga boolean FALSE
pre_page_sga boolean FALSE
sga_max_size big integer 300M
sga_target big integer 300M


2) Problem statement tuning Dev Oracle Server to work with Application server:

Oracle 10G captures snapshots by default every hour. We can take a look at 3 kinds of reports

Go to command prompt under following directory
C:\oracle\product\10.2.0\db_1\RDBMS\ADMIN

sqlplus \ as sysdba

These are the 3 kinds of reports, the first one is very elaborate and it lists out the top queries and also if there is any resource contention.

SQL>@ashrpt

SQL>@awrrpt

SQL>@addmrpt

From looking at the resource contention he identified that it was writing to 3 control files and that is not required for a development database, so we changed to 1 and the log file needed to be increased.

SQL> alter system set control_files='C:\ORACLE\ORADATA\ORCL10G\CONTROL01.CTL' scope=spfile;

SQL> shutdown immediate

SQL> startup

Monday, July 12, 2010

custom component on server start

wc-server.xml need to be updated with the new component:

<Components>
<component compclassname="com.ext.common.configuration.EXTStoreConfigUtilityEnvironment"
enable="true" name="update each environment specific configuration values."/>
</Components>


This is extensively useful for taxware or environment specific components update on each environment. You can load environment specific information on server start. This is loaded into the memory on server start.

Create a new class implementing the ComponentConfiguration:

package com.ext.common.configuration.EXTStoreConfigUtilityEnvironment
public class EXTStoreConfigUtilityEnvironment
implements ComponentConfiguration {

public void destroy() {
//Do Nothing
}

public void enable(boolean enable) throws Exception {
//Do nothing
}
* Main method to perform the action of running all the queries
*/
public void init(Element arg0) throws Exception {

}
}

Saturday, July 10, 2010

Jsession CloneID for debugging

WebSphere Commerce Dyna-caching mechanism only supports cookie-based sessions, and not URL rewriting.
CloneID (also known as server ID) is appended to the session ID

Once a browser establishes session affinity with an application server, a JsessionId is created. When a HTTP session is created, the session id is passed back to the browser as part of a cookie or URL encoding

e.g.

JSESSIONID 00004QkpGtFgJZ01zkgdi7qMJvg:14aelsphk

The value after : is the cloneID of the server and is used to identify the server. It is usually appended to the JSESSION ID.

This is very useful in a clustered environment for debugging and looking into logs.

This CloneID is only visible from /config/plugin-cfg.xml.

e.g. location :
/opt/IBM/WebSphere/AppServer/profiles//config/cells/plugin-cfg.xml

search for CloneID

Wednesday, July 7, 2010

Implementing Stackable product promotions site wide

Out of the box, product promotions can not be stacked, which means only 1 product promotion per item.
Production promotions can be stacked with the following configuration and updates in database. I would strongly recommend testing all different kind of custom and out of box promotions.

Step1 disable the policy 'Product: Any order item can only participate in one promotion' from px_policy

Disabling the policy lets multiple product promotions be evaluated, otherwise just the one with the highest priority goes in (or in the case of equal priorities, whichever happened to be evaluated first.)

update px_policy set status=1 where px_policy_id in (select px_policy_id from px_policy where name like 'Product: Any order item can only participate in one promotion')

update px_policy set xmlparam=replace(xmlparam,'Active','Inactive') where px_policy_id in (select px_policy_id from px_policy wherename like 'Product: Any order item can only participate in one promotion')

Step2: Modify WCSPromotionEnginConfig.xml

Modifying the XML lets promotions stack, instead of just evaluating all of them, and picking the best one.

In the WCSPromotionEngineConfig.xml, you have to disable the behavior true to false This change will allow all the price adjustment based on discounted price (or contract price displayed in the shopcart).

Step3: Update all promtions from accelerator. If you want this to affect your exisiting promotions.

you have to update all the promotions ( don't need to change anything, but need to resave the promotion in order to get promotion XML updated). After that, all the promotions for the same product can be combined.

Saturday, July 3, 2010

Business Auditing feature in WCS

IBM websphere commerce uses : WebSphere Commerce uses the industry standard Triple DES (data encryption standard)

This ensures that a third party using network-sniffing programs cannot snoop on the network when a user submits a password.
Passwords are never decrypted during the authentication process, as is the common security practice.
All user passwords are one-way hashed using the SHA-1 hashing scheme and encrypted using a 128-bit key based on the merchant key.


Business Auditing:

Business auditing is the capturing of the business logic and objects during a WebSphere Commerce operation.A report on business auditing is available in the Administration Console.

WebSphere Commerce business auditing records the information about the execution of business logic, such as the command, request, response, command context, and other information. For example, if a Customer Service Representative overrides a price for a particular ordered item within an order, this needs to be captured, and is useful to resolve any discrepancies between the price quoted to the customer and the customer's bill.

If an error occurs while executing any business logic, WebSphere Commerce attempts to execute the business logic, and this attempt is recorded to indicate that the request failed.

To use the business auditing feature, you must first enable it using Configuration Manager. By default, when you install WebSphere Commerce, the business auditing feature is already enabled. The WebSphere Commerce system captures the execution of specific business logic based on some default configuration. You can further customize which commands you want captured during a business audit (that is, enable existing or add new commands) by configuring the BusinessAuditDataCapture.xml file. The data captured by WebSphere Commerce system is stored in the BUSAUDIT table.


customization: BusinessAuditDataCapture.xml
enabling: For servers it is enabled using the configuration manager

Adding custom commands:
<AuditCommand eventType="MBR" command="com.custom.commands.CustomActivityControllerCmd" audit="true"/>
<AuditCommand eventType="MBR" command="com.custom.commands.CustomOrderSearchControllerCmd" audit="true"/>

Commerce where can I find?

This information below is for WCS 6.0

Accelerator:
https://localhost:8000/webapp/wcs/tools/servlet/ToolsLogon?XMLFile=common.mcLogon&storeId=0

Admin Console:
https://localhost:8002/webapp/wcs/admin/servlet/ToolsLogon?XMLFile=adminconsole.AdminConsoleLogon&storeId=0

Org Admin Console:
https://localhost:8004/webapp/wcs/orgadmin/servlet/ToolsLogon?XMLFile=buyerconsole.BuyAdminConsoleLogon&storeId=0

Dynacache:
https://localhost:8002/cachemonitor/

Nifstack.xml:
\properties\version\update\backup
/opt/IBM/WebSphere/CommerceServer60/properties/version/update/backup/NIFStack.xml

wc-server.xml:

\xml\config\wc-server.xml
WC_LIVE_CELL and WC_LIVE would need to be replaced with the node name.
/opt/IBM/WebSphere/AppServer/profiles/LIVE/installedApps/WC_LIVE_cell/WC_LIVE.ear/xml/config



Temp folder:
\wasprofile\temp
/opt/IBM/WebSphere/AppServer/profiles/LIVE

Logs:
\wasprofile\logs
LIVE and WC_SS_LIVE_1 need to be replaced with appropriate instance names.
/opt/IBM/WebSphere/AppServer/profiles/LIVE/logs/WC_SS_LIVE_1


Tracing:
log into wasconsole
https://hostname:9043/ibm/console
or
Toolkit, right click on the server->goto admin console.

Application servers > Change Log Detail Levels
e.g. after changes done to enable WC_SERVER
*=info:enable.trace.log.*=all:com.ibm.websphere.commerce.WC_USER=all:
com.ibm.websphere.commerce.WC_SERVER=all:
com.ibm.websphere.commerce.WC_ACCESSCONTROL=all

Regular Expressions in Java\WCS Examples

REGEX 101:
Regular expressions are a powerful (and fairly standardized) way of searching, replacing, and parsing text with complex patterns of characters.

. is a wild character
e.g b.t in text could search bat bit bet bot b*t b t

[] search to Limit to a range of characters.

e.g. b[aeio]t : bat bet bit bot

() search more than 1 characters. (can not use [] to search more than 1 character)
e.g. b(a|e|i|o|oo)t : bat bet bit bot boot

* 0 or more times
+ 1 or more times
? 0 or 1 time
{n} Exactly n number of times
{n,m} n to m number of times
- Indicates a range that would match any number\character.
\ used for escaping regexp special characters.
"^" notation is also called the NOT notation. If used in brackets, "^" indicates the character you don't want to match.
\s Search spaces and tabs.
Simpler short cuts:
\d [0-9]
\D [^0-9]
\w [A-Z0-9]
\W [^A-Z0-9]
\s [ \t\n\r\f]
\S [^ \t\n\r\f]

e.g. US telephone number 858-343-1111
Regexp expression to be used: [0-9]{3}\-[0-9]{3}\-[0-9]{4}

If - is optional: 8583431111
Regexp expression to be used: [0-9]{3}\-?[0-9]{3}\-?[0-9]{4}

e.g. 8836KV [0-9]{4} [A-Z]{2}
(first 4 numbers)(last 2 characters.)

e.g. May 11, 2010 : [A-Z]\s+[0-90{1,2},\s*[0-9]{4}
[A-Z]+(first 3 characters)\s+(mandatory space),\s* (0 or more spaces)[0-9]{4}

E.g. These examples are with JDK 1.4.2 API.

import java.util.regex.Matcher;
import java.util.regex.Pattern;

Pattern class:

An instance of the Pattern class represents a regular expression that is specified in string form in a syntax similar to that used by Perl.

A regular expression, specified as a string, must first be compiled into an instance of the Pattern class. The resulting pattern is used to create a Matcher object that matches arbitrary character sequences against the regular expression. Many matchers can share the same pattern because it is stateless.

The compile method compiles the given regular expression into a pattern, then the matcher method creates a matcher that will match the given input against this pattern. The pattern method returns the regular expression from which this pattern was compiled

Matcher Class

Instances of the Matcher class are used to match character sequences against a given string sequence pattern. Input is provided to matchers using the CharSequence interface to support matching against characters from a wide variety of input sources.

A matcher is created from a pattern by invoking the pattern's matcher method. Once created, a matcher can be used to perform three different kinds of match operations:

  • The matches method attempts to match the entire input sequence against the pattern.
  • The lookingAt method attempts to match the input sequence, starting at the beginning, against the pattern.
  • The find method scans the input sequence looking for the next sequence that matches the pattern.

Each of these methods returns a boolean indicating success or failure. More information about a successful match can be obtained by querying the state of the matcher.

This class also defines methods for replacing matched sequences by new strings whose contents can, if desired, be computed from the match result.

The appendReplacement method appends everything up to the next match and the replacement for that match. The appendTail appends the strings at the end, after the last match.


Sample methods :



public static String removeDuplicateWhitespace(String inputStr) {
String patternStr = "\\s+";
String replaceStr = " ";
Pattern pattern = Pattern.compile(patternStr);
Matcher matcher = pattern.matcher(inputStr);
return matcher.replaceAll(replaceStr);
}


public static String replaceSpecialCharsWithHTMLTags(Map splCharsMap, String patternStr, String input)
{
final String METHODNAME ="replaceSpecialCharsWithHTMLTags";

Pattern r = Pattern.compile(patternStr, Pattern.CASE_INSENSITIVE);
Matcher m = r.matcher(input);

m.reset();
StringBuffer sb = new StringBuffer();
String replaceStr = null;
String supTagElement = null;

while (m.find())
{
supTagElement = input.substring(m.start(), m.end());

if (splCharsMap.containsKey(supTagElement))
{
replaceStr = (String) splCharsMap.get(supTagElement);
}

if ( replaceStr!=null && replaceStr.length > 0 )
m.appendReplacement(sb, replaceStr);
else
m.appendReplacement(sb, supTagElement);
}

m.appendTail(sb);
return sb.toString();
}

WCS:

There is application for regular expression in both for front validation JSP validation for feeds and also during string search\replace in the back end.



Ref:
http://java.sun.com/developer/technicalArticles/releases/1.4regex/

Monday, June 21, 2010

Testing with Rspec, Cucumber, Selenium, Watir & Buzz around BDD, TDD

I was lucky to be working with some folks this past week, who are working on a bleeding edge commerce framework and this particular blog is more on the testing frameworks for front end.

I will soon have an article on building a front end store using ROR for Websphere commerce and it will be really useful to automate the test scripts using RSpec and one of the automation tools out there.

This is meant to be 101 for these concepts and I have added links below for deep diving.

BDD and TDD are paradigms essentially bringing the prominence of writing test cases before development and also making it easier on non coders to understand the test cases.

TDD: Test driven development
1. Writing automated test scripts before actually starting to write codes.
2. It is a great idea using any programming language but traditionally development with languages such as Java was pretty high hence it was skipped as the larger development life cycle was blamed for it.
But It makes great sense for using this approach for ROR (Ruby).
3. Test driven development brings lot of clarity to the design in the very early stage of the development cycle.

e.g.

BDD: Behavior Driven Development
1. It extends the concept of the TDD and takes it one level further as it makes writing test scripts in a very expressive way and no programmatic way.
2. It really helps non programmers (BA, QA, PM) to understand the test cases as they are expressed in a simple English like language.
3. Using 'should' when describing the behavior of software to help clarify responsibility and questioning the requirements.
4. Using 'ensure' when describing responsibilities of software to differentiate outcomes in the scope of the code in question from side-effects of other elements of code.
5. Using mocks to stand-in for collaborating modules of code essentially stubbed out without the actual implementation.

e.g. RSPec, Cucumber

Selenium: Selenium IDE and Selenium RC. (Remote Control)

Selenium IDE provides automated way of recoding tests from a Firefox plug in and it has a powerful feature to validate tests by providing several commands to assert the results. It also provides a powerful mechanism to export the test case into a programming language such as Java, Ruby (RSpec),Perl, .etc.

Selenium RC: is a tool that allows you to launch browser sessions and run Selenium tests against those browsers.
To do this, the Selenium RC server acts as an intercepting proxy for the browser

Selenium-IDE does not directly support:

* condition statements
* iteration
* logging and reporting of test results
* error handling, particularly unexpected errors
* database testing
* test case grouping
* re-execution of failed tests
* test case dependency
* screenshot capture of test failures

Flash is not supported as a part of the Selenium-IDE but there is a flash extension for Selenium Flash and it also requires changing the actual Flash action scripts to add methods to be called from Selenium. It is not a very good practice.

The results from the tests can be verified by connecting from the database.

Watir: Watir is automated testing framework. It does not have a IDE of it's own but there is a testwide recorder that can be used. A combination of RSpec and Watir really makes very readable test scripts. It offers Flash support on firefox but I could not find a lot of documentation around this. So I have my doubt's around automating Flash components.

Cucumber is designed to allow you to execute automated integration tests. It is a tool for implementing Behavior Driven Development.It can work in Conjecture with Rspec.

Each feature is expressed as a Give\When\Then

Given: Preconditions to be filled.
When: This section defines what feature does (i.e., the behaviour, the steps).
Then: Testable out come that can be validated.


http://www.oreillynet.com/pub/a/ruby/2007/08/09/behavior-driven-development-using-ruby-part-1.html
http://www.ibm.com/developerworks/web/library/wa-rspec/
http://wiki.openqa.org/download/attachments/400/Selenium+IDE.swf?version=1
http://www.rubyinside.com/cucumber-the-latest-in-ruby-testing-1342.html

Thursday, June 17, 2010

Character Encoding and Commerce

Character Encoding 101:

The "charset" parameter identifies a character encoding, which is a method of converting a sequence of bytes into a sequence of characters. This conversion fits naturally with the scheme of Web activity: servers send HTML documents to user agents as a stream of bytes; user agents interpret them as a sequence of characters. The conversion method can range from simple one-to-one correspondence to complex switching schemes or algorithms...

Reference: Section 5.2 Character encodings of the HTML Document Representation W3C Recommendations

Character encoding tells the browser and validator what set of characters to use when converting the bits to characters.

Types: what does this mean, there are different standards of character set encoding
You see various encoding types

Traditional ASCII is 7 bits and has limited character representation.
encoding="ISO-8859-1"
encoding="windows-1252
encoding="utf-8"

Even though UTF-8 is one of the most popular some of the old browsers might not support hence we used to use ISO-8859-1 encoding in some pages.

windows-1252 :
Windows-1252 is a character encoding of the Latin alphabet, used by default in the legacy components of Microsoft Windows in English and some other Western languages
ISO-8859-1 :
UTF-8,UTF-16,UTF-32 :
Unicode is an international character set standard which supports all of the major scripts of the world, as well as common technical symbols.
Unicode transformation unit and each character is store in 21-bits so depending on each of these UTF types it is stored as a multiple of 8 bytes
e.g. UTF-8 , it could be saved in upto 4 bytes.


Developers:

Java:
In the Java programming language char values represent Unicode characters. Unicode is a 16-bit character encoding that supports the world's major language
UTF-16 (Java).

In WCS:

The language table has a ENCODING column that specifies the encoding for language
e.g.

select encoding from language where language_id=-1;

From end in JSP:
Character encoding

We set the character encoding to UTF-8 as this is what is present in the DB for language encoding.

<%
response.setContentType("text/xml");
response.setCharacterEncoding("UTF-8");
response.setHeader("Cache-Control","max-age=0");
%>


Form Submit: In commerce what ever is specified in front end should match with what ever is specified in the encoding attribute of the language table.

select encoding from language where language_id= language passed in the command context.

Database:
e.g. If you need the character encoding to windows 1252, you can do that in the session while executing SQL statements.

set NLS_LANG=AMERICAN_AMERICA.WE8MSWIN1252

Database Client:

I use today and for the client to show the correct characterset, we have modified.

HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\Key_OraClient...
Change “NLS_LANG” to AMERICAN_AMERICA.UTF8




Good reference URLs:
http://java.sun.com/javase/technologies/core/basic/intl/faq.jsp
http://java.sun.com/docs/books/tutorial/i18n/text/convertintro.html
http://tlt.its.psu.edu/suggestions/international/web/codehtml.html

Sunday, March 7, 2010

JSP Error Handling and Extending OOB framework

Out of the box these 2 are good Articles:

http://publib.boulder.ibm.com/infocenter/wchelp/v6r0m0/topic/com.ibm.commerce.developer.doc/concepts/csdcommanderror.htm

http://publib.boulder.ibm.com/infocenter/wchelp/v6r0m0/topic/com.ibm.commerce.developer.doc/concepts/csdjsperror.htm

Essentially there are following elements:
1. ECSystemException and ECApplicationException classes, that are used to throw exceptions from inside the controller.
2. Extending the ECMessage and ECMessageKey
3. The errors are intercepted by the ExtStoreErrorDataBean and mapped to resource bundles
4. Errors are displayed in JSP.

Define the property in ExtMessages_en_CA.properties (
_ERR_RESTRICTED_PROD = This item is restricted for delivery.


Extending ECMessage and ECMessageKey:

public interface EXTMessageKey extends ECMessageKey{
public static final String _ERR_RESTRICTED_PROD = "_ERR_RESTRICTED_PROD";

public static final String _ERR_MAX_NUM_OF_OF_CARTS_REACHED = "_ERR_MAX_NUM_OF_OF_CARTS_REACHED";

}

public class EXTMessage extends ECMessage{
//this file structure needs to be created inside toolkit properties folder
for com/ext/common/messages/EXTMessages.propertie

public final static String EXT_MESSAGES_RESOURCE_BUNDLE = "com.ext.common.messages.EXTMessages";

// final constants
public static final int USER = ECMessageType.USER;
public static final int SYSTEM = ECMessageType.SYSTEM;

// message severities
public static final long ERROR = ECMessageSeverity.ERROR;
public static final long WARNING = ECMessageSeverity.WARNING;
public static final long STATUS = ECMessageSeverity.STATUS;
public static final long INFO = ECMessageSeverity.INFO;

public EXTMessage(long msgSeverity, int msgType, String msgKey)
{
super(msgSeverity, msgType, msgKey, EXT_MESSAGES_RESOURCE_BUNDLE);
}

//system
public static final EXTMessage _ERR_MAX_NUM_OF_OF_CARTS_REACHED = new EXTMessage(ERROR, SYSTEM, EXTMessageKey._ERR_MAX_NUM_OF_OF_CARTS_REACHED);

public static final EXTMessage _ERR_PROCESSING_REQUEST = new EXTMessage(ERROR, SYSTEM, EXTMessageKey._ERR_PROCESSING_REQUEST);

//user
public static final EXTMessage _ERR_LOGIDPWDPSID_NOT_PRESENT = new EXTMessage(ERROR, USER, EXTMessageKey._ERR_LOGIDPWDPSID_NOT_PRESENT);

public static final EXTMessage _ERR_RESTRICTED_PROD = new EXTMessage(ERROR, USER, EXTMessageKey._ERR_RESTRICTED_PROD);
}


In the controller command, The error needs to be thrown as an exception for example:

throw new ExtendedApplicationException( EXTMessage._ERR_RESTRICTED_PROD,
CLASSNAME,
METHOD_NAME,
ORDER_SHOP_CART_VIEW,
true);

Friday, January 15, 2010

Commerce Developer : Working with ORACLE

As a commerce developer working with Oracle back end, it is very important to have good understanding of SQL.

I am not a DBA, but the way I have learned was to started with basic Insert/Update/Delete, Select commands and using group by, order by
but progressively learned some handy SQL commands.

Refresh local schema for development.
I would use the script below for creating tablespace/schema/assign grants and import schema. Replace wcsatdev with the schema name of your choice.

drop tablespace atdata including contents and datafiles;
drop tablespace atindx including contents and datafiles;

create tablespace atdata
datafile 'C:\oracle\product\ORCL10G\atdata_01.dbf' size 250m reuse autoextend on;

create tablespace atindx
datafile 'C:\oracle\product\ORCL10G\atindx_01.dbf' size 250m reuse autoextend on;

drop user wcsatdev cascade;

create user wcsatdev identified by wcsatdev
default tablespace atdata
temporary tablespace temp
quota unlimited on atdata
quota unlimited on atindx;

grant connect,resource,create materialized view to wcsatdev;

grant create view to wcsatdev;

grant create synonym to wcsatdev;

create or replace directory dpdumpdir as 'C:\projects\db';

create or replace directory dplogdir as 'C:\projects\db';

GRANT READ, WRITE ON DIRECTORY DPDUMPDIR TO wcsatdev;

GRANT READ, WRITE ON DIRECTORY DPLOGDIR TO wcsatdev;

Using data pump in Oracle 10g, faster way to import.
impdp system/oracle@orcl10g dumpfile=DPDUMPDIR:export.dmp logfile=DPLOGDIR:from_wcsatdev_10g.log REMAP_SCHEMA=from_schema:to_schema PARALLEL=8 CONTENT=ALL



When ever you have a larger database on your local database, it is a good idea to run DB stats once in a while to improve performance.

exec dbms_stats.GATHER_SCHEMA_STATS(ownname=>'WCSATDEV',estimate_percent=>dbms_stats.auto_sample_size, CASCADE=>TRUE, DEGREE=>4)

Where degree will invoke the 4 parallel slaves, cascade is required for indexes.

Size of the Tables: Run this command in the schema

select SEGMENT_NAME,sum(BYTES)/(1024) size_in_kil from user_extents where segment_type='TABLE' group by SEGMENT_NAME order by size_in_kil desc;


Dropping Stage prop Triggers: Stage prop is a completely new discussion but on my local I run this for better performance.

set pages 0 lines 100 ;
spool drop_trigger1.sql ;
select 'drop trigger '||trigger_name||' ;' from user_triggers where trigger_name like '%STG_%' or trigger_name = 'STGLOG_STGRESERVED1' ;
spool off;
@drop_trigger1.sql;



Issues and Fixes:

java.sql.SQLException: ORA-01000: maximum open cursors exceeded

check for this. select count(*) from v$open_cursor;

Login with system previleges and type show parameter OPEN_CURSOR

Alter cursors
ALTER SYSTEM SET OPEN_CURSORS=4000 SCOPE=BOTH;

Soln: restart DB


Error: ORA-12514: TNS: listener does not currently know of service requested in connect descriptor.

I made these changes and since then i did not see the problem.
ORACLE_HOME\network\admin\listener.ora, Please look for HOST and change that to
HOST=Windows hostname
ORACLE_HOME\network\admin\tnsnames.ora
For the 10g local instance tnsentry: give localhost instead of the windows hostname
HOST=localhost


Thursday, January 14, 2010

Debug: Generic System Error Test JSP









This would happen due to the descripency between supported languages in wc-server.xml and language table.

_ERR_WRITE_JSP



Wednesday, January 6, 2010

EJB Files to be Checked Into CVS

Creating an EJB bean and best practices to checkin files into CVS for 6.0.

The most important rule, while creating EJB's is to make sure, you understand the data model and finding the composite key that would identify the table.
e.g. extending catentry_ext (catentry_id,store_id), There should be just 1 ejbcreate with both the values.
e.g. extending catgroupdesc_ext (catgroup_id, language_id), ejbcreate(catgroupId,langId)
Also making sure the finders exist with the composite key.
e.g. findByCatentryIdStoreId and findByCatGroupIdAndLangId


I have used the following link to create multiple EJB's successfully.

http://publib.boulder.ibm.com/infocenter/wchelp/v6r0m0/topic/com.ibm.commerce.developer.tutorial.doc/tutorial/ttd18.htm?resultof=%22%62%6f%6e%75%73%22%20%22%62%6f%6e%75%22%20%22%62%65%61%6e%22%20

2. I would usually be careful at step 15. to map the entity bean to the table, some times has issues when you try to map in wrong view. Other than that all other steps are required.

3. Best practice to check in files as a practice. Usually the following files are checked in and an ant task is used during build to generate all the stub classes for EJB's and access beans. It is better to not checkin all the generated classes to CVS. This approach would make it cleaner.
packagename.XXCatEntryExtBean
packagename.XXCatEntryExtKey
packagename.XXCatEntryExtHome
packagename.XXCatEntryExt

WebSphereCommerceServerExtensionsData\ejbModule\META-INF\ejb-jar
WebSphereCommerceServerExtensionsData\ejbModule\META-INF\ibm-ejb-access-bean.xmi
WebSphereCommerceServerExtensionsData\ejbModule\META-INF\ibm-ejb-jar-bnd.xmi
WebSphereCommerceServerExtensionsData\ejbModule\META-INF\ibm-ejb-jar-ext.xmi

META-INF\backends\ORACLE_V9_1\Map.mapxmi
META-INF\backends\ORACLE_V9_1\WEBSPHERECOMMERCESERVEREXTENSIONSDATA_NULLID_XX_CATENTRY_EXT.tblxmi

4.Do not check-in access beans, use a build script to create access beans, when refreshing code from version control system such as CVS or SVN.
To test it, you can use access bean to create a new row and commit.

Interesting things. Why do you use access beans?
Access beans are wrappers around EJB's that providers getters and setters to the fields and are also used to improve the performance of an data update operations, where in multiple columns are written to first Hashmap internally and committed to the database.