SSIS vs DQS Client Cleansing

The first bit of feedback was that interactive cleansing through the DQS Client was known to be faster than SSIS interaction – so my first instinct was to test just how much faster it was – and I was surprised – the speedup was around fivefold. The below chart shows my results for processing 10,000 rows with 1-5 Domains:

SSDQS SSIS vs Client performance

If this scaled, then my 3.9 hour estimate for a 1m row / 10 Domain process would shrink to under an hour. Still not ideal, but getting closer to a production viable speed.

Now, the reason behind this – as explained by the DQS team – is that the component sends discrete chunks through to validate (1000 rows at a time as far as I can tell) which the DQS Server then passes back – which adds overhead and is inefficient for the DQS Server. This is done so that the DQS Cleansing component is not a blocking component. However at this point it’s not possible as far as I can tell – to have any control over the size of these chunks.

Speeding up SSIS processing

The next bit of feedback was to suggest breaking up the work to improve throughput. There’s two ways of doing this – first is to split up the domain processing and second is to break up the data into  chunks and process in parallel. So I tried this by splitting it up the following ways:

1. 5 Domains through a single DQS Cleansing Task – 10,000 rows
2. Each domain though a dedicated DQS Cleansing Task – 10,000 rows
3. 5 Domains through 5 dedicated DQS Cleansing Tasks – 2,000 rows each – 10,000 total

To be honest, the results weren’t overwhelming:

1. Untuned: 94s
2. Separated Domains: 86s
3. Separated Data: 78s

Given that the Separating of Domains means the data would in a real situation have to be split up and recombined, there’s probably not enough saving there to make that approach worthwhile. Splitting up the data yielded a 20% processing time saving – nice, not enough to be really useful given how long it takes normally.

Practical suggestions for the DQS Team

A direct quote from the DQS team’s mail to me was “DQS is designed to best perform on large chunks.”. Looking at the SSIS logs, it’s only sending 1,000 rows at a time – which is clearly sub optimal for DQS + SSIS to interact effectively. So there are two options available for a fix based on my understanding:

1. Make the component configurable to send larger chunks – with a more SSIS like 10,000 rows default
2. Make the component optionally blocking

The first just makes sense and I doubt would be a massive job to make “Rows To DQS Server” a configurable property. The second may be harder – and can probably be duplicated just by setting the new “Rows to DQS Server”  property to zero or a very high number.

In practice it’s still a bit slow for very heavy DW workloads, but hopefully the above suggestion would give it a real boost in performance and make it viable for mid sized ones.

The DQS team have posted about the improvements made in the DQS blog – and the one I really wanted to focus on was performance via SSIS as the CTP3 offering was not viable for large data sets. So this Part 1 post is all about the performance of DQS via SSIS in RC0.

So, I set up a Knowledge Base in the same way as I did for testing CTP3, with 5 duplicate domains – just evaluating an Integer with a single rule saying that integer had to be greater than a value to be valid. Then I ran two sets of values (5k & 10k rows) through the KB via SSIS, evaluating 1,2,3,4 and 5 fields.

So how does DQS Perform?

Here’s the results- the value in the grid is Seconds taken to process.

DQS Performance in SSIS

So – have we moved on from CTP3? A bit. But not much, and enough to be accounted for by a different VM setup (as a reminder CTP3 processing 5k rows took from 20 to 45 seconds for 1-5 columns). I accept a VM may be slower than a properly configured server, but even if it was twice as quick it would still not be a viable option for industrial use.

Looking at execution time changes by number of columns / rows processed, the time taken seems to be pretty linear as rows and columns increase, so it appears DQS performance can be evaluated pretty much as:

DQS Execution Time = Spin Up Time + (Columns * (Rows * Row Process Time))

Where:

Spin Up Time = time taken for DQS engine to start (Constant)

Columns = number of columns being evaluated (Variable)

Rows = number of rows being processed (Variable)

Row Process Time = time taken to process a single row (Constant)

On my VM, Spin Up Time seems to be 7 seconds, and Rows Process Time = 0.0014 seconds.

So, if we had to validate 10 columns on 1,000,000 rows of data (not too crazy) -

DQS Execution Time = Spin Up Time + (Columns * (Rows * Row Process Time))

DQS Execution Time = 7 + (10 * (1,000,000 * 0.0014))

DQS Execution Time = 7 + (10 * (1,000,000 * 0.0014))

DQS Execution Time = 14007 seconds = 233 minutes = 3.9 hours

Which effectively rules it out as a viable production process. Note of course that my formula doesn’t make any allowance for rule complexity.

Is DQS Production ready?

As per anything, the answer is – It depends. For validating small data sets it’s in the realms of slow, but probably acceptable. For big data sets, I’d have to say no – I couldn’t use it in a production environment to validate large sets of data. I’ve added a Connect suggestion to get this on the teams radar.

A practical example of this might be that in a Customer Address record, a missing Postcode might attract a high score as it’s a very important field. However a badly formatted work, home or mobile telephone number may attract a lower score as it may not be as important to the business. Though, cumulatively, if all three numbers are badly formatted that may be necessary to give a combined high score so the record gets examined.

An example of this is below. A failed Postcode gets a score of 3, and a failed telephone number gets a score of 1. Thus, anything with a score of 3 or above either has a failed Postcode, or 3 failed telephone numbers, and can thus be subject to special handling.

Fig 1: Data Quality Scoring Example

From the example above we can see this is a fairly arbitrary process in terms of how scores are calculated and used. SSDQS itself doesn’t natively support assigning a score or weight to a failed data item, but what it does do is provide us with a flexible engine to help us decide what is a failed data item. SSIS can then react to this pass / fail behaviour and apply a scoring.

Setting up an SSDQS Knowledge Base for Scoring

Given that the basis for scoring is pretty binary in nature, I set up a simple KB that had domains that would either pass or fail a piece of data. I first created a data set with three data fields:

• Year – Values ranging from 1970 to 2025
• Value – Values ranging from 0 to 100
• Code – Values A,B,C,D,E

I then set up a KB to evaluate the fields as follows:

• Year – Valid from 1975 to 2020
• Value – Valid from 10 to 95
• Code – Valid values A,C,E

Note that I did not set up any Domain Values or do any training – I just set up the KB, Domains and Domain Rules. All I want to use DQS for is to identify records that are invalid for SSIS to use in scoring.

Using SSIS to Score SSDQS output

Next I hookup up my SSIS Data Quality Cleansing Component to push the source data through the Knowledge Base, and get the status of each of the columns after they pass through. As there are no preloaded valid values in the Domains, the status comes back as either “Invalid” (it failed the Domain rule) or “Unknown” (in this configuration, this translates to a correct value).

DQS output Data Viewer

The Data Quality Cleansing Component doesn’t support scoring in itself. This has to be added using a Derived Column on an item by item basis. Using a simple IF / THEN / ELSE expression, I assign a score of 1 to each failed column based on the status of the record, as below:

Applying Score using a Derived Column

Because of the Pipeline nature of SSIS, I then need to add a second Derived Column transform downstream to weight and add the scores together to create a final, record level score:

Aggregating and Weighting Score using a Derived Column

This results in a final Data Quality “Score” assigned to each record:

Weighted Scoring output Data Viewer

What you then do with these scores is up to you. In my example package, I used a Conditional Split to send records with a score over a certain threshold to a different destination:

DQS Scoring example Data Flow

Improving the Scoring process

The example I’ve created is quite simplistic – it has hard coded weightings and redirection thresholds, and can only react to two (of a possible three) record statuses.  The process could be made more flexible using metadata driven weightings and thresholds (provided as package inputs).

Beyond that you have the option to handle the clean and dirty data more appropriately – by pushing dirty data into a cleanup process, halting ETL processes etc, etc.

The key takeaway here is that DQS enables you to create a scoring process that is independent of the actual Data Quality rules that pass or fail a piece of data. The DQS Knowledge Base is your flexible input of what qualifies as a good or bad record, instead of having to hard code using SQL or Derived Columns, which could get messy very quickly.

By default SSIS Script Tasks / Components reference .NET Framework 2.0, which was confusing me as I tried to implement a solution around managing times in different Time Zones which required the TimeZoneInfo Class which only exists in 3.5 and higher. (This was what I was trying to implement: Daylight Savings in Script Task). Because the script referenced .NET Framework 2.0 the TimeZoneInfo class wouldn’t work. Switch the target framework to 3.5 and bingo – all good.

The tl/dr version: In your script component – while editing the script – open the Properties and change the Target Framework to 3.5 to use functions available in 3.5.

What does this mean?

1. First up this only applies to SQL Server connections – other OLE DB connectivity such as Teradata and Oracle will be unaffected.
2. In the short term, nothing – if you are using OLE DB it will carry on working for a fair few years yet – but if you are planning for a systems longevity – look to using ODBC for all your SQL Server connectivity needs

It’s a slightly confusing move given that ODBC underperforms OLE DB – so it’s bad news for us SSIS people, unless part of the change also includes some significant performance and capability improvements.

The performance profile is as below:

SSIS DQS Component Performance

There seems to be a fairly linear relationship between the number of domains being processed and execution time. Note that I’ve created a dummy value for “0″ to indicate what the start-up time of the DQS component might be, as it’s impossible to have a DQS Cleansing Component in the flow with no columns mapped.

I’d ignore the actual numbers – this is on a development VM which is definitely not configured for performance – and I’m aware the DQS Team are working on performance issues (though by the looks of it, better be working hard).

The DQS Cleansing Component

So, in our (shiny, new) SSIS Toolbox we have a new component to connect to DQS – the DQS Cleansing Component:

SSIS DQS Cleansing Component

The DQS cleansing component pushes a data flow to the DQS Engine for validation. This requires a special Connection Manager, the DQS Cleansing Connection Manager, which as we can see below is a simple creature:

SSIS DQS Cleansing Connection Manager

The sole option at this point is to choose which DQS Server to point at. So, lets look at what we get in the SSIS Component once we use the Connection Manager:

SSIS DQS Cleansing Component Connection Manager options

Once again – still nice and simple – choosing your Connection Manager allows you to then pick from a list of Published Knowledge Bases. Once a KB is selected, a list of the available Domains is populated, though there is nothing you can do with this list other than review it. So next we move to the Mapping tab:

SSIS DQS Cleansing Component Mapping Tab

The usual suspects are there – pick your input columns in the top half of the tab and they become available for mapping in the lower half. Each input column can be mapped to a single Domain (I can’t quite see how Composite Domains work in this context). You then get three output streams – the Output, Corrected Output and Status Output. The Output is just the column passed through, Corrected is the column value corrected by the DQS Engine and the Status is the record status (which comes out as Correct, Corrected or Unknown which corresponds to the DQS Data Quality Project statuses. In the Advanced Editor you can also switch on Confidence and and Reason Outputs, which relate to matching projects.

Note that there is only a single output for the DQS Cleansing Component – if you want to send OK, Error and Invalid records to different locations, you will need to do so with a downstream Conditional Split component.

Summary

So we’ve had a quick look at the basics of automating DQS activities using SSIS, and how SSIS plugs in to the DQS Server. Subsequent posts will start digging into some practical implementation including performance.

Some further reading can be found here:

• Using the SSIS DQS Cleansing Component – from the DQS Blog

Whilst this is slightly disappointing, fortunately for those who like the Dimension Merge (formerly Kimball SCD) component, Pragmatic Works have come to an arrangement with Todd McDermid to include a supported version of it available in a future release of their TaskFactory bundle of SSIS components. This means for those who live in risk-averse enterprises, you can now use the component as it comes with a support package.

The Components

I’ve had a scan through the Tasks in the Control Flow and Components in the Data Flow to see what’s there. First observation is that all the components have got funky new icons. I don’t see anything new function-wise on the control flow, but it’s interesting to note the Execute DTS 2000 Package task remains, implying this piece of backwards compatibility will still remain. 

SSIS Data Correction Component

There’s only one completely new item available on the Data Flow, which is the Data Correction component. However as per this blog post by Hennie de Nooijer on his BI Future blog, it appears to be unusable. It looks like this is where SSIS will touch upon the Data Quality Services component of SQL Server (SSDQS), but since it has no Advanced Editor available and it can’t be configured by the normal editor there’s not much I can find out about how this component works. One for review in a future CTP. 

There’s a sort of new, sort of not, item in the form of the Source and Destination assistants, which are effectively Wizards for building Source and Destination  adapters. It tidies up the toolbox a bit but other than that doesn’t seem to add much for experienced users. 

Fix these problems!

However, there are things that desperately needed to change that haven’t. The Slowly Changing Dimension component has had no visible changes, leaving us with the poorly performing (usually to the point of unusability) RBAR toy we have already (a previous suggestion on Connect was killed in 07). The Pivot Transformation still has no usable interface (a previous suggestion on Connect was killed in 06) and remains confusing and difficult to configure. The Derived Column transform still locks you into Unicode strings if you have a text field, and overrides your setting if you change it, then alter the expression, and there also appears to be no new functionality in terms of added functions (my request to make this extensible on connect has been killed). 

There generally seems to have been a lack of updates that address some of the existing issues with individual components – given that some of these – especially the SCD and Pivot – have been poor since day 1 – now 5 years ago – it’s a shame to have seen no changes at all. 

To at least get the Pivot and SCD fixed I’m raising new connect items for them – please vote for them by following the links below and we may get them fixed in a future CTP: 

• Request for a decent UI for the Pivot
• Suggestions to improve the SCD component

Thanks in advance!