dtexec: Basic Package Execution

Like dtutil, dtexec uses a similar set of basic options to identify where the package you are operating on is located: /File, /DTS & /SQL. The eagle eyed among you will note that while the full option name is the same the abbreviated is slightly different, though their operation is the same. There is also a slight difference in the /SQL additional options – because no movement of the package is taking place, you only need to specify the /Server, /Password and /User for the package being executed (rather than separate identification of the Source and Destination package). Similarly there is a /Decrypt option for providing the password to encrypted packages. For details on this, check either my previous post or MSDN documentation.

There are two important options around basic execution. First is the /X86 option. This forces SQL Agent to to execute the package in 32-bit mode on a 64-bit server, which can be important in scenarios where 64-bit compatibility isn’t available. This commonly is an issue with drivers – however this is ignored when running from the command line. When running scenarios where you have SSIS servers in your environment running in 32 and 64-bit you probably want to read this MSDN article first: 64-bit Considerations for Integration Services.

The second is the the option to /Validate the package instead of executing it. What this does is halt the package after the validation phase – this can be useful for testing in deployment scenarios. How I sometimes use it is following deployment with a dtutil script, I run a dtexec script to validate the deployed packages, which helps catch any issues with missing / incorrect configurations etc. before you do a test run. This can be used in conjunction with the /W[arnAsError] option, which will treat a warning as an Error and cause validation to fail.

As a side note, you can place comments in the dtexec command line using the /Rem option.

dtexec: Configuring Execution

Execution can be configured using a number of options. First up is the ability to get dtexec to reference a file with all your dtexec options set in it, using the /Com[mandFile] {filespec} option. This can be handy if you are in an environment where you call dtexec via a remote tool such as Control M that has a character limit in terms of the batch command it can send.

You can provide or override the configurations used in execution using the /Conf[igFile] {filespec} to point at an XML configuration file. Similarly you can override specified connection settings using the /Conn[ection] id_or_name;{connection_string} option – multiple instances of this option are allowed and you can either point at a connection name or the GUID of the connection. You can associate an SSIS logging type with package execution using the /L[ogger] classid_orprogid;{configstring} option – this could be useful in troubleshooting if you don’t want logging enabled during normal execution but want to be able to turn it on for debug.

Individual Package properties such as variable values can be /Set – and you can set as many as is required. Finally you can set the Maximum number of executables permitted using the /M[axConcurrent] {concurrent_executables} option.

dtexec: Using Checkpoints

dtexec can make a package override its defined checkpoint usage settings – though as i’ve alluded to before, be very careful when using checkpointing as you won’t always get the expected results. But the options available are to turn /CheckPointing {on\off}, specify the actual file using /CheckFile and specify the usage of Checkpoints at package start using /Restart {deny | force | ifPossible}. These options match the settings in the package of SaveCheckpoints, CheckpointFile and CheckpointUsage.

dtexec: Execution Logging and Reporting

When running packages from dtexec, you get some additional logging options which can be useful when you are trying to do a more detailed debug on a package. Some are useful for immediate visualisation in the console window as you execute while others create file based logs.

The first console option is /ConsoleLog, which allows you to write to the console window the same entries that would normally be captured by SSIS execution logging. You can select which of these columns you want to capture, the most important of these likely to be Message. Further fine tuning in the settings for this option allow you to tighten the logging to a specific task or event – again as per standard logging. Next is /Reporting  level, which tunes what type of event to capture – Errors, Information, Warnings and so on. Usually when you want this level of reporting i’d advise using the Verbose switch to just capture everything. There is also the capability to create an event exclusion list for this option. Finally, /Sum will put a count of the number of rows the next component will receive in the log.

If you want to push all this logging out to a file (and the logs can get pretty big so they can blow out the console window pretty quickly) you can write them out using the /VLog option. If you need to venture further into debugging you can create the dump files .mdmp and .tmp using either the /DumpOnError option to dump on any error or /Dump with a specified error to only generate the files under certain circumstances. These are advanced options and to understand the output i’d advise reading the MSDN article Working with Debug Dump Files first.

dtexec: Package Verification

The dtexec utility offers a few options to verify the package you are pointing at is the one you intend to execute. First up is the option to verify the build versions using the /VerifyBuild option which allows you to tell dtexec to only run the package if it has the specified Major (and optionally Minor) build version. These are the values set at the control flow level in the SSIS package you are calling. You can check the package has a valid digital signature using /VerifySigned option. Finally are the options to match the GUID of the package you are executing using /VerifyPackageId and the version GUID of the package using the /VerifyVersionID option.

All of the options above are interesting additional protections to ensure your system is running the code that it should be, but realistically they are poor and fiddly substitutes for a proper release management process.

Why use dtexec?

Most of the options above are of course covered in SQL Agent execution options, but there are scenarios where dtexec can prove useful – most commonly where SQL Agent isn’t available or isn’t permitted to be used by IT policy. It also has value in deployment and testing scenarios. Most of the time I would advocate using SQL Agent to schedule and automate your jobs, but it’s good to have such a flexible tool available.

There is also a GUI option, dtexecui.exe which allows for the configuration of almost all the above options using a simple GUI. It also has the advantage of being able to generate the dtexec command line prompt as well, making syntax errors much more avoidable.

Official MSDN documentation can be found here for 2008 and here for 2005.

Lunchtime Wed 3rd March, 2010 – Ensuring Optimal Performance in SQL Server 2008 Based Applications with Viktor Isakov

Evening * Thu * 4th Mar 2010 – What’s new in Reporting Services 2008 R2 and PerformancePoint Services 2010 with Peter Myers, presenting what’s new in the upcoming release of Reporting Services 2008 R2 and PerformancePoint Services 2010

And TBA date in April 2010 – Knights of the SSIS Round Table – Kevin Wong, Glyn Llewelyn and myself will be presenting a series of mini demos followed by Q&A, so a chance to pick some expert brains

Hope to catch you at one of the sessions!

Why Migrate from Cognos to Microsoft?

There are myriad reasons why anyone may want to move from Cognos to Microsoft, but i’ll probably surprise you by saying functionality isn’t one of the bigger drivers. Each suite has its own strengths and weaknesses and on balance there’s probably not enough strength on either side to move for that alone, unless addressing specific business needs. I’d view the main reasons as:

1. Licensing- Cognos is a suite of separate applications that need to be licenced separately, as opposed to SQL Server which is a single product. There can be significant cost and administrative savings from moving to the Microsoft suite. How much this works out to be will depend on your licensing agreements, but reality is Microsoft is the cheaper option.
2. Existing SQL Assets – Many times companies have SQL Server licences for their databases and want to extract more value from those licences
3. Cognos 8 upgrade- Your business may be looking at the cost of moving to Cognos 8 from Cognos 7 and not seeing enough ROI potential.
4. Performance- many Cognos users report unsatisfactory performance from deployments
5. Uncertainty- With the IBM buyout of Cognos, the number of products likely to be in the mix has gone up – what’s going to stop being supported? What other tools will I need? With Microsoft you have the certainty of a single integrated platform with a long and clear roadmap
6. Skills availablity – there are simply more Microsoft BI developers out there, and generally they cost less to employ and are cheaper to train.
7. Application Integration- Microsoft applications are quite open in terms of connectivity, customisation and integration with other applications. Cognos is a closed box . For example anyone can talk to an Analysis Services cube, but no non-Cognos application can talk to a Cognos cube. Reporting services can easily be extended with custom controls.

Decision Stream / Data Manager = SSIS (Integration Services)

Decision Stream (Cognos 7) and Data Manager (Cognos – though there is minimal difference between the two – are Cognos’ ETL tools. They are showing their age badly, and are very “black box”. I personally loathe the clunky interface and untunable performance. But – they work. The SQL Server equivalent tool is Integration Services – a much more up to date, faster and configurable ETL tool. SSIS is a more generalist tool compared to Data Manager, and so it requires a bit more thought (though not a lot) for building Data Warehouses. However if you have large volumes of data to shift, Cognos simply cannot compete.

The reality is ETL is one of the hardest components to migrate. It often has a lot of business rules (and in Data Manager it’s possible to hide them in a million different places) and if you get the migration wrong, you stand to devalue your existing DW. In any migration scenario this would likely be the last on the list, unless addressing a specific need. I would generally leave this as a legacy system that gets superseded over time.

PowerPlay = SSAS (Analysis Services)

Cognos cubes are built in PowerPlay in both Cognos 7 and 8 (in fact there was no upgrade of Powerplay in Cognos 8). Whilst adequate, there are many stories of business migrating to SSAS for significant performance benefits – I am aware of one client whose cubes processing time went from 24 hours to 30 minutes by migrating. SSAS has significant market penetration – getting hard figures is difficult (without paying a lot of money – last public figures are to 2006) but Microsoft is the leading vendor by a significant margin – so obtaining expertise is less difficult (though as with ETL, true OLAP wizards are rare and expensive creatures).

Cubes are one of the easier components to migrate (unless you have very complex ones), as any well designed cube is usually sat on top of a robust ETL which handles the really complex business rules. In the best case scenario you just have to build your dimensions and measures and the job is done.

Report Studio = SSRS (Reporting Services)

Technically speaking, Visual Studio is the direct equivalent as that is the design environment, but Reporting Services is SQL Servers’ Reporting engine. It provides fixed parameterised reporting with the capacity to provide automated deliveries through easily managed dynamic lists. SQL2005 was functional but a little lacklustre, but in SQL2008 has come on leaps and bounds. It’s not the most stellar part of the Microsoft BI stack, but flat reports don’t usually need stellar functionality.

Migration of reports is usually pretty straightforward – you have the layout and formats already – and once the underlying cubes are set up it’s simply a matter of rebuilding each report.

Query Studio = SSRS Report Builder or Excel

Query Studio is a user tool for building simple reports, functionality which is carried out in the SQL world using a tool called Report Builder, or even just though Excel (see the section below on Analysis Studio). Personally, I don’t like Report Builder, though the next edition (3.0) is supposed to be an improvement – but then so was 2.0, so I’m a little sceptical. However if you have good cubes, or have access to the forthcoming PowerPivot, most analysts who want ad hoc data will migrate rapidly to Excel.

This is unlikely to be a significantly used application so the change in functionality probably won’t be an issue, but here Cognos has a better offering.

Analysis Studio = Excel

For working with cubes from an analytical context, Excel is Microsoft’s tool for the job. Using functionality based on Pivot Tables you get a capable (though admittedly not perfect) tool for drilling down, slicing and performing analysis on OLAP data. Unless you have very demanding users, Excel will meet most needs – especially if you have upgraded to 2007 (and if you haven’t, you really really should). If you do have demanding users, there’s a host of 3rd party applications which plug in to excel and can dress specific needs.

Further to this basic functionality, workbooks with connections to SSAS can then be uploaded to SharePoint and distributed via a set of technologies called Excel Services, allowing for fast dissemination of results.

Metric Studio = SharePoint

 Metric Studio is aimed at Dashboarding and Scorecarding. Microsoft at one point had an offering called PerformancePoint to address this, but it suffered a premature demise, but the relevant components migrated into SharePoint, and anyone with an Enterprise licence can install them. This isn’t a particularly exciting tool for either suite so I’ll move on. The migration path is similar to that to SSRS.

Cognos Connection = SharePoint

Cognos Connection is the BI portal for the Cognos suite. Sharepoint is the same for the Microsoft stack… and  an Enterprise CMS to boot, plus a whole heap of other things I have only limited awareness of. Cognos Connection will have better integration with the BI Metadata, Sharepoint will have better integration with your Enterprise generally. It depends what matters more to your business.

Migration here depends on the extent to which you have SharePoint deployed in the organisation already. If it’s there, it becomes a manual exercise in migrating content. If its not, then it’s a major enterprise change.

Bits that don’t match on either side

Of course there are areas where the two suites don’t quite tie up. Cognos has Planning, which Microsoft did, but now doesn’t. Microsoft has PowerPivot, an in memory analysis tool which Cognos hasn’t got a counter offer for. Framework Manager has an approximate equivalent in the Data Source Views that underpin SSAS cubes, but the implementation is quite different and more database oriented. Content Store, the Cognos metadata store has no match in the Microsoft stack and can be perceived as one of the MS BI stacks bigger weaknesses. Event Studio is approximately matched by SQL Server Notification Services. Of course Microsoft’s whole underpinning is based on SQL Server, a robust enterprise database system with scaling for massive warehouses and Master Data Management which Cognos has no direct offering for.

Final Comments

Undoubtedly i’ve got a few things wrong here – my knowledge of the Cognos stack is not as detailed as that of the Microsoft Stack – and I’ll welcome any corrections from the Cognos Community. This post is intended to help those with Cognos installations understand what Microsoft has to offer, the applications involved and what they translate to in Cognos speak. Some of the features I mention are due in SQL2008R2 which will be released in a few months.

If you are looking for an answer to the question “Should I Migrate?” then the answer will always be that it depends on your circumstances. As I said at the outset, each stack has its own strengths and weaknesses and each one may address your business needs better than the other. This is something that needs careful analysis from experienced BI professionals. I look forward to the debate this is likely to promote!

/* Apply Multiple LIKE clauses in a single WHERE clause */
SELECT *
 
FROM [AdventureWorks].[Person].[Contact]
 
WHERE CASE
  WHEN FirstName LIKE ‘Gusta%’ THEN 1
  WHEN FirstName LIKE ‘Cath%’ THEN 1
  END = 1

An elegant solution!

There are two that impact SSIS – dtutil and dtexec. In simple terms, dtutil moves packages, and dtexec executes them. In this post I will be focussing on dtutil. dtexec will be covered in a later post.

dtutil: deploying packages

The main use of dtutil is to script the deployment of packages. For file operations it is equivalent to a simple copy operation, though you can add SSIS specific operations, such as changing the package GUID. However it also allows for movement to and from the SQL Server msdb store, which can’t be done via copy / paste.

Deploying a package requires you specify its source location using one of the following options:

• /Fi (or /File) + File Path + Package Name.dtsx – for packages in the File System
• /DT (or /DTS) + Package Name – for packages in the SSIS Package Store
• /SQ (or /SQL) + Package Name – for packages in msdb

There are a few important things to note about the above options. Firstly for File System operations, you need to specify the .dtsx extension. Secondly, for SSIS Package store operations you don’t need to, even though it is still strictly speaking a file system operation. Finally, you don’t need to know where the Package Store is – dtutil works that out for you, which is handy if you have custom setups of the package store.

For packages located in msdb, you may also need to provide connection details using the following options:

• /SourceS (or /SourceServer) – the source server name. If not used, it assumes localhost
• /SourceU (or /SourceUser) – the SQL Authentication user name
• /SourceP (or /SourcePassword) – the SQL Authentication Password

Note that the username and password options are only required if SQL Authentication is being used. If left out, dtutil will try and authenticate to msdb using Windows Authentication for the current user. 

Next up, you need to tell dtutil to copy the package and specify the destination. The option to move it is /C or (/Copy). This is then followed by the same options as specifying the source - though without the preceding backslash (i.e. Fi, DT or SQ) – then a semicolon, and finally the destination path. If you are using a SQL Server destination you may also need to provide connection details with the /DestS, /DestU and /DestP options.

As usual, an example is worth its weight in gold, so heres some to help you understand. In each case I will give the Full and Abbreviated versions – both of which do exactly the same job.

Moving a package from File to SSIS Package Store:

dtutil /File C:\Package.dtsx /Copy DTS;Package

dtutil /Fi C:\Package.dtsx /C DT;Package

Moving a package from msdb on a Named Instance of SQL Server using SQL Server Authentication to the file system

dtutil /SQL Folder\Package /SourceServer SQLSERVER.INSTANCE /SourceUser Monkey_User /SourcePassword P@$$word /Copy File;C:\Package.dtsx

dtutil /SQ Folder\Package /SourceS SQLSERVER.INSTANCE /SourceU Monkey_User /SourceP P@$$word /C FI;C:\Package.dtsx

Using a little batch scripting these can easily be converted into jobs which will deploy a bundle of packages quickly and easily.

dtutil: altering packages

dtutil is more than a deployment tool – it can also alter packages using the following options:

• /Encrypt – copies and encrypts the package with a specified password and protection level
• /IDRegenerate – Generates a new GUID for the package
• /Sign  – Applies a digital signature to a package

The last two can be useful as options within the deployment process, where the Encrypt can replace the Copy operation if you want the deployed environments packages to be more secure.

dtutil: file and folder operations

Finally, dtutil can work components of the file system or MSDB folder structures, creating and deleting files and folders

• /Delete – Delete a package
• /Exists – Test if a package exists
• /FCreate – Create a folder
• /FDelete – Delete a folder
• /FDirectory – List the contents of a folder
• /FExists – Test if a folder exists
• /FRename – Rename a folder
• /Move – Move a package from one location to another

Most of these would be useful in an initial deployment or for tidying up.

dtutil: other things

There are a few leftover functions which I don’t have a categorisation, so here they are.

• /Quiet – suppress prompts
• /Remark – add a comment
• /Decrypt – decrypt a package as an operation is performed

Why use dtutil?

The main reason for using dtutil is simply that the BIDS supplied deployment options, and all that mucking about with Deployment Manifests, is limited, not as scriptable, and also has been buggy, having problems with configuration files (though I believe this is now fixed in 2008, but since I stopped using it long ago, I can’t be sure). dtutil also offers additional functionality, such as being able to script package operations such as encryption.

Official MSDN documentation can be found here for 2008 and here for here for 2005.

Enter… BIDS Helper

Over at codeplex there is a project called BIDS Helper, which adds some useful additional functionality to the standard BIDS environment.  It improves the development environment for integration, analysis and reporting services.  In this post I will focus more on the SSIS capabilities that are added.  The additional functionality is fully documented on the codeplex site, and I will highlight some of the more useful functions below – the links are to the documentation which is nice and visual, making it easy to understand.

Expression and Configuration Highlighter - this adds a highlight to components, connection managers and variables that are controlled by an expression and / or configuration, which allows you to rapidly see what is subject to change at runtime.

Expression List – this lists every property that is set by a variable (though currently not expressions that set the value of variables) – great for finding out what is actually in the package, as these things can be easily overlooked otherwise

Non-Default Properties Report – this lists any property in the package that is set to a non-default value. Very handy for debugging and seeing what is non standard.

SSIS Performance Visualization- this executes the package and presents a gantt chart of the execution time, and also can present performance over time – useful when performance tuning.

Variables Window Extensions - this adds the functionality of being able to change a variables scope – very handy for when you create a variable at the wrong scope (something I do all the time).

BIDS Helper is a handy little tool that adds a few nice functions for SSIS developers without intefering with the existing UI or design environment – I thoroughly recommend installing it, especially as it’s free!

Enter SQL Server Auditing, introduced in 2008.

What is SQL 2008 Auditing?

Well, you could read the official documentation on MSDN, but it’s a little overwhelming, so here’s the view from 500 ft. SQL Server Audit provides a secure means of tracking access and changes made to the database schema or its data. What this means is you can trace who tried to access what and when. If any command was issued to change the data, you can track this as well. You cannot track the changes made to the data. So for example if someone made an insert into a table, you could track that someone made an insert, who did it, when that insert was made and what the SQL of the insert statement looked like – except for what data was inserted. That task falls to Change Data capture (CDC) which isn’t directly tied to Auditing (currently).

How do I implement Auditing?

A SQL Server Audit is made up of two components, the first of which is the SQL Server Audit. This is a server level object (created in the master database) that defines where logs will be written. Logs can be written to either a file, the Application Log or Security Log. These must be created before you create any actual Audits.

The second component is either a Server Audit or Database Audit. Server Audits again exist at the server level, and track activities that occur at the server level, such as login attempts or permission changes. Database Audits exist within an individual database and track activities at a database level such as schema changes or data operations.

You can have multiple SQL Server Audits defining multiple log target locations. Multiple Server and Database audits can be created to use a given SQL Server Audit. This probably makes more sense when explained in pictures:

Fig 1: SQL 2008 Auditing Hierarchy

So how do I audit my SQL Agent Jobs?

Setting up Audits can either be done through SQL Server Management Studio (SSMS) or through SQL scripts. As usual any action carried out through the Wizard, so I will do the wizard first, then show the generated scripts.

Step 1 is setting up the SQL Server Audit. To create this in SSMS, under the Security Node of the Server is a folder called “Audits”. To create a new Audit, simply right click on the folder and choose “New Audit..”, as set out below:

Fig 2: Creating a SQL Server Audit

Then in the Wizard just enter the Audit name and in the dropdown select “Application Log”, as shown below. The alternatives are to the Security Log or to a File, but I won’t be covering those in this simple example. Click on OK and you’re done.

Fig 3: Creating a SQL Server Audit

The equivalent SQL script is below:

USE [master]

GO

CREATE SERVER AUDIT [My Sample Audit]
TO APPLICATION_LOG
WITH
( QUEUE_DELAY = 1000
 ,ON_FAILURE = CONTINUE
)

GO

Now you have somewhere to write your log. Note that by default SQL Server Audits are disabled when they are created, which means nothing will be written to your log until until it is enabled. This can be done just by right clicking on it in SSMS and choosing “Enable”.

The next bit is to set up the Database Audit on the SQL Agent Jobs table, which is in the system table [msdb].[dbo].[sysjobs]. Under the msdb database, open up the “Security” folder and within there is the “Database Audit Specifications” folder. Right click here and choose “New Database Audit Specification…” as shown below.

Fig 4: Creating a Database Audit Specification

This will open the Create Database Audit Specification dialog. Because we are operating on msdb, unless you are logged in as sa, you probably won’t be able to create objects and the process will fail. So ensure the user you are logged in as has appropriate permissions on msdb. Because I’m playing on a Dev environment, I just allowed my own user access to msdb and gave them db_owner rights – this probably won’t fly in a production environment, of course!

The dialog is shown below. First task is to choose which SQL Server Audit to write to – so we use the one we created above called ”My Sample Audit”. You choose the Audit Action Type – i.e. what you want to log. In our case we want to see when changes are made to our jobs in SQL Agent, so we choose the “UPDATE” Action Type. Next we have the Object Class, which for this case is a Database Object, so “Object” is selected, and then the Object Name. Finally the Principal defines who is Audited – in the example I have selected “public” because every user is in the public role, so I will capture any users UPDATE commands issued on my selected table.

Fig 5: Creating a Database Audit Specification

Now here we hit a minor hiccup – you will note in the example above I have selected the object sysdtslog90 – it’s the only non-system view in the database, and the only one I can get to come up in the browser. The solution here is just to hit the “Script” button, and modify the generated code to point at the right table, as shown below:

USE[msdb]

GO

CREATE DATABASE AUDIT SPECIFICATION [SQL Agent Audit]
FOR SERVER AUDIT [My Sample Audit]
ADD (UPDATE ON OBJECT::[dbo].[sysjobs] BY [public])

GO

This code runs just fine, and as you will see has the desired effect. As with SQL Server Audits, Database Audit Specifications are created in a disabled state. As before, just right click to enable and the Auditing will begin.

Finally to test, just change the Enabled state of any SQL Agent job on your server, then check the Application Log, where you will find an Event detailing who just changed its status!

Bitlocker Drive Encryption encrypts the entire disk transparently. What “Transparent” really means to me and you is that if you copy a file from a Bitlocker Encrypted disk to an unencrypted disk, that file is readable by any machine. However should someone try to read the encrypted disk directly they wouldn’t see anything except encrypted garbage. So if your disks are Bitlocker Encrypted, unless someone has the Recovery Key, your drives and data are unreadable.

How do I enable Bitlocker?

So how does this work in practice? Well, a lot depends on whether your computer has a TPM 1.2 chip on the motherboard. If you do, you can simply open the  Bitlocker control panel (just type Bitlocker in the search bar) and enable Bitlocker. The TPM chip manages the storage of the Bitlocker keys – and Windows 7 will stop the process if you don’t have a key. You set a PIN and after that your computer won’t boot without a PIN, and if someone tries to remove the drives to read them – well, they can get the drives out but reading theier content is impossible.

If you don’t have a TPM chip, you can put the Bitlocker key on a USB Key. To do this requires a bit of a workaround as by default Bitlocker will only work if you have a TPM chip installed, and the wizard will block progress if you don’t. Fortunately the nice folks over at sevenforums.com have posted a guide on How to Turn BitLocker On or Off without a TPM for Windows 7. It involves going into the Group Policy Editor (some sort of security thing) but is incredibly simple to do. Again, not 100% sure why Microsoft decided to make this a bit awkward to set up, so hopefully this will change in the future.

You can then apply Bitlocker and store the key on a USB drive. From now on your computer will only boot if you have the USB Key plugged in (you can – and should - remove it after the computer has booted). This is a minor inconvenience and you have to remember to keep the USB key seperate from the machine – otherwise if someone steals your computer and you’ve left the USB key in, your encryption efforts are wasted.

How do I prevent me locking myself out with Bitlocker?

This applies to those of us using USB Keys, but the advice is simple:

• Keep multiple copies of the Bitlocker recovery keys somewhere accessible and safe – such as on a external HDD, Live Mesh, a webmail account…
• Create multiple USB keys – from the Bitlocker control panel you can load the startup key on as many USB keys as you like. I have 3 copies so if a USB drive fails, or goes walkies, I don’t have to worry about not being able to use my computer. You can also store multiple computers bitlocker keys on one USB key.

I don’t have Ultimate / Enterprise – what are my options?

Well, option 1 is to upgrade using Windows Anytime. But assuming you don’t want to, the other option is Folder Level encryption. All you need to do is pick a folder, right click, select Properties and under the Advanced options choose to Encrypt the folder. If this is the first time you are encrypting something on your machine it will prompt you to back up the security certificate – as with the Bitlocker recovery keys I strongly recommend making multiple copies in multiple safe locations . Your folder will then encrypt and turn green so they are easy to spot. They aren’t angry, like the Hulk

Now this isn’t transparent – if you move a file from your encrypted folder to another location it remains encrypted and unreadable without the security certificate. But if someone stole your PC, unless you have autologin set, those files are unreadable, which for the purposes of this discussion is what we intended

So, am I safe and secure now?

Well, maybe, as long as you have done a few other basic things:

1. Require a password on startup
2. Have a screensaver that requires a password to unlock
3. Backed up recovery keys
4. Backed up your data
5. Backed up your data
6. Backed up your data

Good luck, and keep your data safe!

A BI Monkey update

The BI Monkey has been a little short of access to computers for a while, as burglary took his laptop, being on holiday in Japan and Hong Kong distracted him and just to add some extra confusion into the mix, I switched consultancies towards someone a bit more Microsoft focused. However I am almost back in the saddle, having finally got SQL2008 up and running on 64-bit Windows 7. Once i’ve dealt with the vast amounts of admin that seem to accompany joining my new company, i’ll carry on with my quest to cover every component in depth. However I may end up delivering all my new samples on a 64-bit architecture – though i’m still looking at setting up a virtual PC to stick with 32-bit SQL2005 for creating my demo files.

None of this seems to have slowed down the growth of the site, though it is starting to calm down – bimonkey.com had it’s first 5,000 hit day earlier this month and will easily top 100,000 hits this month. This makes me very happy! Now, if only the Google ad revenue would come pouring in!

Windows 7 first impressions

My new work machine is loaded with Windows 7 Enterprise, and I have to say, i’m impressed. I wasn’t much of a Vista hater so perhaps I am an easier sell, but there are a few features that really stand out for me:

• Taskbar – the new smaller icons are nice and the way you can manage multiple windows from a single icon is very slick
• Libraries – Being able to collect several disparate folders under a single master folder makes managing files much easier
• Snap – a simple but easy way of pulling windows to cover half the screen for side by side review is handy
• Speech – I was a fan in Vista, and whilst not much has changed, it does seem better behaved – so will be dictating more in future

The introductory videos here are a great and easy introduction to these features, and I reccommend all new 7 users take a peek.

As the requirements were scoped out, it became apparent that the resultant Data Warehouse was going to be a very large and complex thing that would take a long time to develop. We did a quick pilot to demonstrate that the ETL could deliver results in a structured and controlled manner into the Warehouse, which delivered about 5% of the files through from end to end. IT were pleased as it all went well. Their first thought was then to deliver the remaining 95% of data.

Do you need to put all the Data in the Warehouse?

I asked the client to pause for thought at this point. In delivering the 5% of data we had proved that the DW process was feasible to IT – but had given nothing to the business. We had solved an IT problem, not a business one. In order to get the business excited we needed to deliver someting to them. The IT side of the client saw the wisdom of this – if the business had something tangible and helpful it helped sell the project. Also, if they hated it, we could find out we were on the wrong track in a far cheaper manner than after delivering a monster DW that nobody wanted!

So what did we do next? We focused on satisfying just one business problem – in this case the data needs of a single department, and set about building the parts of the DW required to meet those needs. At the end of this we could throw them a shiny new cube and see if it stuck. This way we would:

• Deliver initial results at minimum cost
• Establish if we were on the right track from an early stage
• Get business visibility of the DW faster

If all went well, we could focus on delivering the next 5%, then the next…  and probably find that there was probably a big chunk that wasn’t used anyway, thus giving more savings – or identifying areas for new insight. Much better that delivering the big bang DW that may actually be a damp squib!

So was that Agile BI then?

It was a form of it, yes! The idea of Agile Development is to deliver results quickly, allowing the business to see what they are getting sooner and tune the results. Now, this isn’t all good – it can lead to massive scope creep, drift of purpose and even failure to deliver anything at all. So there is a need to be very cautious and apply this methodology only where appropriate. Cases such as Cubes for analysis and Reports are things that can benefit from a rapid development cycle carried out in close conjunction with the business. A good example of where not to apply it is the ETL – it’s something that needs rigour and structure to be delivered effectively, and in practice is something business users neither understand or care about.

Sometimes with BI it is the right approach in greenfield sites because of one significant issue - clients can have a poor grasp on the capability of BI solutions and need leading into the wonderful world of BI. Often they can be skeptical about the promises made by consultants and by using an Agile approach on the first build you can get them into the BI mode of thinking rapidly. The client sees what they are getting and can get excited – whereas with standard waterfall delivery  they can wait months to see results after an long analysis / design phasse, and if the consultants have misjudged the business needs – kill the DW in its infancy.

But how can we deliver BI without the Warehouse?

I’m now on another project which has become lost in the design phase and the client is wondering what is going to come out of it all. We shifted approach to an Agile method where instead of focusing on getting the solution letter perfect in Design (Waterfall style) we are going to deliver a functioning SSAS Cube based on a sample of real data and build the design off of that. This way the client gets to see the light at the end of the tunnel and we know we are building the right solution even before the design is finalised.

The key in this case is to establish what the end deliverable is as quickly as possible and demonstrate that we, as consultants, have identified the right target in order to give the business confidence. The DW can follow to support that deliverable.

So when should a project go agile?

There’s no hard and fast rule, but I would suggest the following indicators may guide you towards an initial Agile build:

• Requirements are sparse / undefined – and the goalposts can move
• It is the first BI project and BI capabilities are poorly understood
• Business users are highly available
• You need to give confidence you can deliver the right thing

In case you want to know more, here are a few additional articles on Agile BI that I have found:

• Agile Best Practices for Data Warehousing (DW)/Business Intelligence (BI) Projects from Scott W. Ambler @ agiledata.org
• Agile Development for BI from BI Guru
• Architecting an Agile Data Warehouse from Lee Arnett @ information-management.com
• Agile Data Warehousing: Delivering World-Class Business Intelligence Systems by Ralph Hughes – a comprehensive book on the subject