Is ETL Development doomed?

A slightly dramatic title, but over the last few months I’ve been exposed to a number of tools that will provide a strong layer of automation to ETL development, eliminating a lot of the basic need for ETL developers to shift data from System A to Database B and beyond.

I’ve been hands on with a couple:

And also heard about a couple more:

… and I’m certain this list is not comprehensive. The significant takeaway is that software build automation in the BI world is starting to catch up with where other software languages have already been (Coded a website lately? many IDE’s do most of the hard work for you now). Much as IDE driven tools such as DTS, SSIS and so on moved us away from hand coding SQL and wrapping up those scripts, the latest round of tools are moving us away from the IDE’s where we drag and drop our flows.

How will ETL careers be killed?

There seems to be a couple of tracks for this. First is the pure development automation tools, such as Varigence MIST. If you are technically minded, take a look at this product demo video – though I suggest skipping to about 25 minutes in to see the real meat as it does go on a bit. It looks mindbogglingly powerful but is clearly shooting at the ETL pro who wants to churn stuff out faster, more consistently and with less fiddling about. MIST is limited to SSIS/AS (for now) and I’m not sure how far it will go as it’s clearly aimed at the developer pro market, which is not always the big buyers. I expect to be playing with it more over the next few weeks on a live project so should be able to get a better view.

The second path appears to be more targeted at eliminating ETL developers in their entirety. AnalytixDS wraps up metadata import (i.e. you suck in your source and target metadata from the systems or ERWIN), do the mapping of fields and apply rules, then “push button make code”. Obviously there’s a bit more to it than that, but the less you care about your back end and the quality of your ETL code (cough Wherescape cough) the more likely this product will appeal to you. Say hello, business users, who are the big buyers (though I look forward to troubleshooting your non-scalable disasters in the near future).

What’s the diagnosis, doctor?

Long term, the demand for ETL skills will decline on the back of these tools. Simple ETL work will simply go away, but the hard stuff will remain and it will become an even more niche skill that will pay handsomely – though you may spend more time configuring and troubleshooting products than working with raw code. Which class of tool dominates is uncertain, but I’m leaning towards the business oriented mapping tools that completely abstract away from ETL development altogether.

If you’ve had any experience with these tools, I’d love to hear about them in the comments.

8 thoughts on “Is ETL Development doomed?

  1. TimeXtender is also such a piece of software. Haven’t worked with it myself, but some colleagues have and apparently they like it a lot.

    Personally I use BIML through BIDSHelper, but fiddling around with XML isn’t exactly peachy (and very developer oriented).

  2. Hi James,

    Nice article, and thanks for mentioning BIReady!

    The impending doom of the ETL developer is something I have seen coming for some years now, as you will already know. The fact of the matter is that business users can no longer afford to wait 3, 6 or 12 months for an information system.

    The problem as i have seen it is that most DW developers aim to deliver a star-schema, which, as you know, is not a trivial task – especially if you have multiple data sources. But on they go anyway and end up with a solution that is hard to maintain or change – and all DW projects are a journey, not a destination.

    Bill Inmon gave us the way with Third Normal Form (and Dan Linstedt more recently with Data Vault). Either of those architectures should be applied BEFORE building your star-schemas. The debate has raged for years over who is right – Inmon or Kimball. The reality is they are both right. Incidentally, Bill Inmon recently joined BIReady’s Advisory Board, so we’re doing something right.

    We are pleased to say that BIReady gives the benefit of all. It will design and build a Third Normal Form or Data Vault architecture for you, and then build your star-schema data marts using a wizard. In doing so it will crush months of work into a few days.

    In that respect, we believe our Data Warehouse Automation tool is a lot more automated than the others you mention(!). We are always ready (BIReady?) to put our money where our mouths are and always up for a challenge to deliver something fast and complete.

    If you would like to try BIReady for yourself, drop me a line.

    Best regards,

    Ian Nicholson
    BIReady Australia

  3. Thanks for the responses gents.

    Koen, yes I’ve heard of TimeExtender before – my understanding (though it is dated) is that like WhereScape, it does the job, but the generated code is a bit basic – which is ok for many situations but not all.

    Ian, will reach out to you separately – from your description BiReady is more aligned with AnalytixDS in the breadth of capability. However (and I’ve probably had this conversation with you before) – the ETL coding itself is only a part of a DW build – data analysis, business rule definition – the understanding of the data – that’s the hard part which these tools can’t (currently) address. Not that this diminishes the value of these tools in accelerating delivery, of course.

  4. ‘BI Automation’ tools are nothing new, they have been around for years. If they were going to obsolete ETL developers, they would have already done so by now. To use your analogy of Web Development, sure the tools have moved on markedly, but… well as we all know web developers aren’t obsolete. And you *seriously*, more then ever, need to know how to code as a web developer.

    It’s the same in ETL – I’ve used many of these products (including a few of the ones you’ve mentioned) and are using one now… and worked developing automated BI tools like this. These are great tools for automating repetitive tasks, and I love them but I wouldn’t go too far – like the IDE’s and drag and drop tools of web development they aren’t going to obsolete ETL developers.

  5. ETL development will never be dead. There will always be one offs that will break the automated tools and there will always need to be someone who works on maintaining the plumbing. That being said ETL development is changing as a result of automation. As automation increases the role of a well formulated architecture will become more and more important.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>