SSIS vs DQS Client Cleansing

The first bit of feedback was that interactive cleansing through the DQS Client was known to be faster than SSIS interaction – so my first instinct was to test just how much faster it was – and I was surprised – the speedup was around fivefold. The below chart shows my results for processing 10,000 rows with 1-5 Domains:

SSDQS SSIS vs Client performance

If this scaled, then my 3.9 hour estimate for a 1m row / 10 Domain process would shrink to under an hour. Still not ideal, but getting closer to a production viable speed.

Now, the reason behind this – as explained by the DQS team – is that the component sends discrete chunks through to validate (1000 rows at a time as far as I can tell) which the DQS Server then passes back – which adds overhead and is inefficient for the DQS Server. This is done so that the DQS Cleansing component is not a blocking component. However at this point it’s not possible as far as I can tell – to have any control over the size of these chunks.

Speeding up SSIS processing

The next bit of feedback was to suggest breaking up the work to improve throughput. There’s two ways of doing this – first is to split up the domain processing and second is to break up the data into  chunks and process in parallel. So I tried this by splitting it up the following ways:

1. 5 Domains through a single DQS Cleansing Task – 10,000 rows
2. Each domain though a dedicated DQS Cleansing Task – 10,000 rows
3. 5 Domains through 5 dedicated DQS Cleansing Tasks – 2,000 rows each – 10,000 total

To be honest, the results weren’t overwhelming:

1. Untuned: 94s
2. Separated Domains: 86s
3. Separated Data: 78s

Given that the Separating of Domains means the data would in a real situation have to be split up and recombined, there’s probably not enough saving there to make that approach worthwhile. Splitting up the data yielded a 20% processing time saving – nice, not enough to be really useful given how long it takes normally.

Practical suggestions for the DQS Team

A direct quote from the DQS team’s mail to me was “DQS is designed to best perform on large chunks.”. Looking at the SSIS logs, it’s only sending 1,000 rows at a time – which is clearly sub optimal for DQS + SSIS to interact effectively. So there are two options available for a fix based on my understanding:

1. Make the component configurable to send larger chunks – with a more SSIS like 10,000 rows default
2. Make the component optionally blocking

The first just makes sense and I doubt would be a massive job to make “Rows To DQS Server” a configurable property. The second may be harder – and can probably be duplicated just by setting the new “Rows to DQS Server”  property to zero or a very high number.

In practice it’s still a bit slow for very heavy DW workloads, but hopefully the above suggestion would give it a real boost in performance and make it viable for mid sized ones.

One of the topics we got on to was niche marketing – as this is exactly what they have done – pick a niche, focus on it and get good at it to build a good brand. No different to what Avanade do – their niche is certain areas of Enterprise IT (which admittedly is a whacking great big niche) but it’s an area of focus all the same.

Recruiter Keyword Spam makes me do what?

If you have no niche or market specialisation then you will be unfocused on your customers or market and this will become apparent when you deal with either. This manifests in recruitment with our friend keyword spam. I have a bit of SAS experience on my CV – which I last actively used in 2001 and am quite clearly – based on experience and work history over the last decade - a Microsoft / Enterprise BI guy now, and have no interest in SAS jobs. Nor would anyone hire me for one. But still I get spammed by unfocused recruiters who keyword search and send out jobs without accounting for my experience and skills. What impression does this give me? That as a recruiter you don’t know what you’re doing either in respect of me or your client. What behaviour does this drive? It drives me to add you to my junk mailer list and thus I will likely never deal with you again.

Conversely – if you know your market and your customers – any interaction will be beneficial, or at least likely enough that as a vendor your customer will think you input is at least worth taking the time out to hear. If the BI & DW guys called I’d answer because I know they would have something interesting to offer. Most recruiters go straight to voicemail and then the trash.

So apologies for a bit of a rant – but bad marketing annoys me – particularly when it’s aimed indiscriminately at me.  In summary – go join the BI & DW LinkedIn group - and don’t offer me SAS jobs.

The DQS team have posted about the improvements made in the DQS blog – and the one I really wanted to focus on was performance via SSIS as the CTP3 offering was not viable for large data sets. So this Part 1 post is all about the performance of DQS via SSIS in RC0.

So, I set up a Knowledge Base in the same way as I did for testing CTP3, with 5 duplicate domains – just evaluating an Integer with a single rule saying that integer had to be greater than a value to be valid. Then I ran two sets of values (5k & 10k rows) through the KB via SSIS, evaluating 1,2,3,4 and 5 fields.

So how does DQS Perform?

Here’s the results- the value in the grid is Seconds taken to process.

DQS Performance in SSIS

So – have we moved on from CTP3? A bit. But not much, and enough to be accounted for by a different VM setup (as a reminder CTP3 processing 5k rows took from 20 to 45 seconds for 1-5 columns). I accept a VM may be slower than a properly configured server, but even if it was twice as quick it would still not be a viable option for industrial use.

Looking at execution time changes by number of columns / rows processed, the time taken seems to be pretty linear as rows and columns increase, so it appears DQS performance can be evaluated pretty much as:

DQS Execution Time = Spin Up Time + (Columns * (Rows * Row Process Time))

Where:

Spin Up Time = time taken for DQS engine to start (Constant)

Columns = number of columns being evaluated (Variable)

Rows = number of rows being processed (Variable)

Row Process Time = time taken to process a single row (Constant)

On my VM, Spin Up Time seems to be 7 seconds, and Rows Process Time = 0.0014 seconds.

So, if we had to validate 10 columns on 1,000,000 rows of data (not too crazy) -

DQS Execution Time = Spin Up Time + (Columns * (Rows * Row Process Time))

DQS Execution Time = 7 + (10 * (1,000,000 * 0.0014))

DQS Execution Time = 7 + (10 * (1,000,000 * 0.0014))

DQS Execution Time = 14007 seconds = 233 minutes = 3.9 hours

Which effectively rules it out as a viable production process. Note of course that my formula doesn’t make any allowance for rule complexity.

Is DQS Production ready?

As per anything, the answer is – It depends. For validating small data sets it’s in the realms of slow, but probably acceptable. For big data sets, I’d have to say no – I couldn’t use it in a production environment to validate large sets of data. I’ve added a Connect suggestion to get this on the teams radar.

SAP also offers plenty of BI tools as part of it’s offering – arguably too many. What’s less known is it’s quite possible to overlay Microsoft BI over the top of it. After all, an ERP system is still just a source of data, and BI is all about making sense of data. How practical is it? Well, I know that at Avanade we’ve done one of the world’s biggest implementations. And here is a Microsoft case study where the same has been done.

So the obvious question then becomes – why bother? And there’s two very clear drivers for that:

• User empowerment
• Cost

To extend on each – User Empowerment comes from the higher ease of use that MS tools offers, the self service capabilities through PowerView & PowerPivot, SharePoint and our old friend Excel. Which of course then leads to greater adoption and a more successful move to a data driven organisation.

Cost – well, SAP ain’t cheap. Even with the costs involved in building an alternative solution (and there are people out there who can give you a head start) – Microsoft’s licencing model means that you can reach more users at a lower cost.

So now you know.

It’s small and simple but has the important feature for me: it supports Domain based Windows Authentication, which means I can connect to my corporate domain using my Windows log on and download. Other download managers struggled with this. Plus, no adware.

Topics that he covered included:

• ACID vs BASE (i.e the battle between consistency of data vs. availability of data)
• NoSQL is a means of querying raw data with no cleansing / structure / ETL
• His expectation is that Structured (SQL) and Unstructured (Hadoop) data will coexist in organisations
• Hadoop consists of Storage (HDFS) and Process (MapReduce)
• MapReduce is too complex to work with so languages such as Hive and Pig sit on top of it
• Sqoop is the tool to make Unstructured and Structured data talk – but performance is not good

I can’t really do his talk justice but now I understand Hadoop a whole lot better – essentially it’s just a read only store of unstructured data, a very different beast to a relational database and addressing totally different needs.

I took away the following interesting points from that session:

1. CDC (Change Data Capture) is supported more effectively through some new components – a CDC Control, CDC Source and CDC Splitter
2. ODBC improvements mean improved performance for non SQL Server databases
3. Connection Managers get a few new features – Offline, Expression and Project indicators. Also Offline Connection Managers are picked out through a timeout and importantly now halt validation of any related components (so you no longer get those drags as SSIS tries to validate components and flows hooked to dead connections)
4. File Connection Manager can now handle variable numbers of columns (i.e. it won’t crash)
5. Pivot gets a UI – hurrah! (Note: SCD still sucks)
6. Project Parameters can be configured at design time though Visual Studio Configurations
7. Breakpoints are now in the Script Component so we can see what data is causing or components to blow up
8. Data Taps – data viewers for live execution that dump out to .csv files
9. Package Execution via PowerShell or even T-SQL!

Matt also gave a preview of Barcelona in action, and it looks pretty neat.

I also attended a DQS session that showed a few new features on the UI. Elad Ziklik highlighted that the CTP3 release should be viewed more as a capability preview rather than a test drive of a functional product – so looking forward to a new CTP.

One more day down – one to go…!

So, here are my takeaways from day 1:

From the keynote:

Denali becomes SQL2012 and is slated for release in the first half of next year (way to allow yourself some leeway on the final release date!)

Crescent becomes Power View – and will work on multiple mobile platforms – Apple & Android via Browser, WP7 via a richer app

HADOOP is going to be supported in Windows server – the first CTP is due next year. If you have access to the PASS DVD’s / Sessions – see Dr Dewitt’s presentation on HADOOP – very enlightening

From wandering around the summit:

As per Elad Ziklik of the DQS team, performance is one of their focuses for the next release

I got to finally meet Ivan Peev of CozyRoc after many years of email and phone conversations, and he told me about their new SAS Connector – which allows reading and writing to SAS datasets without an actual SAS install.

I also got to meet Matt Masson, guru of the SSIS team who told me about Project Barcelona – a tool that will do data lineage, metadata management and impact analysis via a crawler as opposed to a manually maintained set.

I also got to sit in on a customer feedback session about BI in the Cloud – unfortunately all under NDA – but it was a great forum to discuss and help direct Microsoft’s Cloud BI ambitions.

I also had a chat with fellow Aussie BI guy Roger Noble who told me about a use for the Term Extraction transformation in SSIS – using it to scan through documents and auto-tag them as the were uploaded to SharePoint – which is pretty cool.

So, that was Day 1… Day 2 to follow!