ODM’s Achilles’ Heel
Alas, even the mighty harbor weaknesses and vulnerabilities. And Decision Validation Services, IBM’s Operational Decision Manager’s test tool, is its Achilles’ heel.
The importance of rule testing cannot be overstated. Business rules are the very heart of key decisions that operate your business and it is imperative that these business rules be thoroughly tested prior to production deployment. Rulesets should be tested independent of the applications code that invokes them. Rules should be validated by business users without reliance on IT staff. Cognizant of this fact, Decision Validation Services (DVS) is supplied as an Operational Decision Manager (ODM) module that enables ruleset testing for developers, business users, and QA engineers. Testing using DVS typically involves creating scenarios (in Excel) that represent real or fictitious use cases to validate the behavior of your rules along with the expected results. These scenarios can be executed directly from Decision Center which then generates a nifty report identifying the status of each scenario, as shown in the sample below.
At first blush, the ability to easily create and execute test scenarios seems wonderful and the fresh scented promise of empowerment holds exciting allure to a business user. But does DVS live up to its promise and potential? Let’s explore…
The Promise
DVS can play a pivotal testing role through various releases of a rule application. During inception, it can be used to validate the initial prototyping activities. Beyond that, business users can continually build scenarios to validate (unit test) business rules.
Over the course of time, sufficient scenarios are built up that they can provide a means to run regression testing. This is very important because as we know, the only certainties in life are death, taxes and model changes. Regression testing is important whenever technical refactoring or tuning is performed.
As is depicted above, DVS has the potential to be of valuable use for business users and by IT staff through the entire lifecycle of a rule application.
The Reality
As is often the case, reality intrudes.
To begin with, a DVS scenario is spread across multiple tabs in a spreadsheet. One needs to navigate through these tabs to piece together a complete scenario. For even a moderately complex domain with a number of DVS tabs, one soon get the disorienting feeling like chasing the scenario down a rabbit hole, while losing track of where you’ve come from. Now say you’ve assiduously navigated the tabs and located the record you want to update – can you? Not so fast! Since a record on a worksheet may be reused by multiple scenarios, you may be inadvertently impacting other scenarios.
Model changes, and the attendant refactoring, is simply an important fact of life that every software professional must acknowledge. There are several types of refactoring that can invalidate existing DVS assets. Some of these changes, such as changing dynamic domain values, can even be initiated by business users with no involvement of IT. This makes it even more critical for business users to have the refactoring tools. However, DVS makes no allowance for this. There are no automation tools that can “migrate” a DVS scenario file to an updated data model. Manually migrating by copying and pasting rows to the new scenario template is unfeasible because much of the DVS Excel data is hidden (hidden rows, hidden properties, hidden tabs). Bluntly put, the reality is that prior DVS assets cannot be used after any non-trivial business object model changes.
These are not the only obstacles to using DVS. To varying degrees of frustration, business users are frequently stumped by these questions:
Invalid Excel data leads to errors that are detected only during test execution. How do I validate my input data?
The data relating to a scenario spans across multiple tabs. How can visually see which data rows are used in which scenarios and how can I easily navigate across tabs for a specific scenario?
How do I detect and cleanup orphan records?
Can I change the data in a record without impacting other scenarios?
When the data model changes, how can I apply these changes to existing DVS files?
Over time, as I create several DVS files, how can I reorganize the scenarios contained in them?
How to I determine which rules are not yet tested by my test suites?
During test execution, I’m encountering an exception. What should I do about this?
How well has my rule been tested? How many different scenarios execute my rule?
There has been a major change due to evolution of the business object model or performance tuning. How can I run all prior DVS files to ensure that the business functionality has not been impaired?
The reality of using DVS is that it faces challenges - some crippling, some mere inconveniences – at every stage of the rule application development, as depicted below.
The Shield
At RuleScape, we decided to do something about these challenges and restore the shine on DVS. Specifically, we built two tools – DVSBoost and DVSBatch.
DVSBoost
DVSBoost is an Excel add-in which complements DVS by overcoming each of the previously mentioned challenges during scenario development and management. Once installed, DVSBoost attaches to your Data ribbon and presents you capabilities on working with DVS scenario files.
It provides a number of critical capabilities:
Sophisticated refactoring support: Automagically migrates your existing DVS files to an updated business object model.
Navigation: On each worksheet, creates hyperlinks on all relevant cells to navigate to its definition.
View data dependencies: On each worksheet, creates comments on object labels that identifies which scenarios it participates in.
Viewing of complete scenarios: View a report with holistic view of selected scenarios.
Validation of scenario input data: Identify format and other issues and suggest fixes.
Management of existing DVS assets: Allow for extraction and merging of scenarios across different DVS spreadsheets.
DVSBatch
DVSBatch addresses the execution challenges of DVS. It executes some or all of your DVS scenario files and reveals insightful execution details that help you devise better test cases and identify areas which need more testing. In addition, it helps you leverage your DVS test scenarios as a means of regression testing whenever a major change has been made to the rule foundation.
DVSBatch is packaged as an Eclipse plug-in Test Accelerator which generates the preconfigured, easy-to-use Java projects for you to execute the tool. Using the tool is as simple as placing your DVS scenario files in a configurable folder and running the prepackaged launcher.
At the core of the functionality is the insightful Coverage Report generated by DVSBatch which provides rich details relating to how well your tests cover your rules. This report lists out the scenarios testing any particular rule, rules that have not been tested and other revealing statistics. This helps you to decide which rules now need to be your focus of testing.
Together, DVSBoost and DVSBatch unlock the full promise and potential of DVS. With these tools, your business users can truly take control of rule validation and your IT staff can confidently perform the necessary refactoring or tuning of the application.
For more information on DVSBoost and DVSBatch, check out our Tools page.
