However, where the Microsoft BI stack has a competitor beating edge is the 3rd Party ecosystem – so where there is a gap in the toolset, often another company will step in and fill it. In this case, Data Lineage issues are addressed by a tool created by Pragmatic Works (which is run by Brian Knight, an SSIS heavyweight) called BI Documenter.

BI Documenter Review

So what does the tool do? It has 3 key functions:

Documentation Generation: The tool auto-generates documentation for Databases, SSIS packages, SSAS Cubes and SSRS reports. It’s quick, and the output is pretty – and it’s really a bit useless. Its benefit is if you have to say you’ve produced some documentation and need to do so with minimal effort. The reason I say this is because it provides no context for why things have been done, what the purpose of the component is, where it fits in to the framework etc. My usual gripe with documentation I come across is that it only tells me the what, not the why, and the why helps me solve a problem. This tool can’t do anything to address this.

Data Lineage: Now this is where the main value lies. Through simple navigation you can select any object (table / view / package etc) in the solution and see what objects depend on it and what objects it depends on in turn. This is great in a complex system where if you need to make a change and need to find out what that impacts.

Now its not perfect – it seems to skip documenting some sources, such as flat files, so they get missed in the impact analysis. And the level of granularity is at the object level – for example you can’t see the impact of an individual column change, just at the table level, but its still a great start and a useful tool.

Snapshot comparison: A final piece of value which can be useful in troubleshooting. BI Documenter takes snapshots of your solution to document – and you can compare these to identify changes in the solution. The detail level is pretty good and will be a great place to start tracing changes in your system when your source control systems fail.

Conclusions

Is the tool worth it? At a maximum cost of US$500 a seat, it’s definitely got a place somewhere in your organisation. The documentation tool is of limited use but the Data Lineage and Snapshot comparisons are worth the cost of the product. Full details here: Pragmatic Works – BI Documenter

The design choices in question are around the managing of history data. In one component of the system it relates to Database storage design, the other relates to Cube storage design. In both cases the history data is stored in a separate location to the “current” data.

Databases: Why separate history tables are a bad idea

The first – and most compelling – reason for not storing your history data in separate tables to your current tables is that it increases complexity for users. Instead of having one location to look for data, your users now have to use two.

The second compelling reason is that there is no point to doing this from a storage point of view. SQL 2005 & 2008 (Enterprise editions only, admittedly) provide partitioning. This enables the contents of an individual table to be stored in separate locations on different filegroups. This means that you can store your current days data in one location and your history in a different one. The reason for doing this is the same as splitting it into separate tables – that querying the current section will be faster than the historic section.  In theory queries against partitioned tables should in fact be faster as the current data is now no longer in the same filegroup as the history data.

Now, there is an overhead associated with designing and maintaining partitions but I don’t see that it is significantly larger than that required to deal with the process required to archive data into separate tables on a daily basis. Additionally when maintaining separate history tables, you need to separate out every single table, whether it gets 10 rows a day or 10 million. With partitioning you can just target the large tables that need that focus.

There are other downsides to maintaining separate tables. If you make a change to a table design, you need to do it in 2 places.  You also need to remember to update your history processes. If your history process fails, you can end up with users getting unexpected query results or ETL process failures when the system loads the next day’s data into the current tables –  and untangling it becomes a real mess. If your partition processes fail to run, you just have too much data in one filegroup for a while – unlikely to be fatal.

So if you have large tables you need to split out for performance purposes – do it at the back end, using the power of the database – which is designed to store data efficiently. Keep it away from the users – they neither need to know or care about your need to keep the data separate. If you want to give them a single object to query with the current day’s data, just use views.

Cubes: Why a separate history Cube is a bad idea

Much of the above applies here – SSAS also has partitions – so you can again store your historic and current data in separate physical locations with the users being totally unaware of this. Again there is overhead in maintenance, but this will also balance out with the maintenance and risks associated with maintaining two identical cubes that only differ in terms of data source.

Use your storage options!

So without banging on about the same things any further, please consider the following two points whenever considering managing your history data:

1. How does what i’m planning affect my users?
2. How does what i’m planning leverage the platforms capabilities?

An Alias is an Operating System level setting that allows you give a friendly name to your Server – and you can then connect to the server using that friendly name. What this means is that as you migrate your code from environment to environment, if you use the same friendly name for your SQL Server in each tier of the deployment, you don’t have to change your connection strings.

It’s a neat trick (though i’m not sure how the names resolve if you use the same name in environments that can see each other) – it certainly requires some proper planning from an infrastructure point of view.

Lunchtime Wed 3rd March, 2010 – Ensuring Optimal Performance in SQL Server 2008 Based Applications with Viktor Isakov

Evening * Thu * 4th Mar 2010 – What’s new in Reporting Services 2008 R2 and PerformancePoint Services 2010 with Peter Myers, presenting what’s new in the upcoming release of Reporting Services 2008 R2 and PerformancePoint Services 2010

And TBA date in April 2010 – Knights of the SSIS Round Table – Kevin Wong, Glyn Llewelyn and myself will be presenting a series of mini demos followed by Q&A, so a chance to pick some expert brains

Hope to catch you at one of the sessions!

Enter SQL Server Auditing, introduced in 2008.

What is SQL 2008 Auditing?

Well, you could read the official documentation on MSDN, but it’s a little overwhelming, so here’s the view from 500 ft. SQL Server Audit provides a secure means of tracking access and changes made to the database schema or its data. What this means is you can trace who tried to access what and when. If any command was issued to change the data, you can track this as well. You cannot track the changes made to the data. So for example if someone made an insert into a table, you could track that someone made an insert, who did it, when that insert was made and what the SQL of the insert statement looked like – except for what data was inserted. That task falls to Change Data capture (CDC) which isn’t directly tied to Auditing (currently).

How do I implement Auditing?

A SQL Server Audit is made up of two components, the first of which is the SQL Server Audit. This is a server level object (created in the master database) that defines where logs will be written. Logs can be written to either a file, the Application Log or Security Log. These must be created before you create any actual Audits.

The second component is either a Server Audit or Database Audit. Server Audits again exist at the server level, and track activities that occur at the server level, such as login attempts or permission changes. Database Audits exist within an individual database and track activities at a database level such as schema changes or data operations.

You can have multiple SQL Server Audits defining multiple log target locations. Multiple Server and Database audits can be created to use a given SQL Server Audit. This probably makes more sense when explained in pictures:

Fig 1: SQL 2008 Auditing Hierarchy

So how do I audit my SQL Agent Jobs?

Setting up Audits can either be done through SQL Server Management Studio (SSMS) or through SQL scripts. As usual any action carried out through the Wizard, so I will do the wizard first, then show the generated scripts.

Step 1 is setting up the SQL Server Audit. To create this in SSMS, under the Security Node of the Server is a folder called “Audits”. To create a new Audit, simply right click on the folder and choose “New Audit..”, as set out below:

Fig 2: Creating a SQL Server Audit

Then in the Wizard just enter the Audit name and in the dropdown select “Application Log”, as shown below. The alternatives are to the Security Log or to a File, but I won’t be covering those in this simple example. Click on OK and you’re done.

Fig 3: Creating a SQL Server Audit

The equivalent SQL script is below:

USE [master]

GO

CREATE SERVER AUDIT [My Sample Audit]
TO APPLICATION_LOG
WITH
( QUEUE_DELAY = 1000
 ,ON_FAILURE = CONTINUE
)

GO

Now you have somewhere to write your log. Note that by default SQL Server Audits are disabled when they are created, which means nothing will be written to your log until until it is enabled. This can be done just by right clicking on it in SSMS and choosing “Enable”.

The next bit is to set up the Database Audit on the SQL Agent Jobs table, which is in the system table [msdb].[dbo].[sysjobs]. Under the msdb database, open up the “Security” folder and within there is the “Database Audit Specifications” folder. Right click here and choose “New Database Audit Specification…” as shown below.

Fig 4: Creating a Database Audit Specification

This will open the Create Database Audit Specification dialog. Because we are operating on msdb, unless you are logged in as sa, you probably won’t be able to create objects and the process will fail. So ensure the user you are logged in as has appropriate permissions on msdb. Because I’m playing on a Dev environment, I just allowed my own user access to msdb and gave them db_owner rights – this probably won’t fly in a production environment, of course!

The dialog is shown below. First task is to choose which SQL Server Audit to write to – so we use the one we created above called ”My Sample Audit”. You choose the Audit Action Type – i.e. what you want to log. In our case we want to see when changes are made to our jobs in SQL Agent, so we choose the “UPDATE” Action Type. Next we have the Object Class, which for this case is a Database Object, so “Object” is selected, and then the Object Name. Finally the Principal defines who is Audited – in the example I have selected “public” because every user is in the public role, so I will capture any users UPDATE commands issued on my selected table.

Fig 5: Creating a Database Audit Specification

Now here we hit a minor hiccup – you will note in the example above I have selected the object sysdtslog90 – it’s the only non-system view in the database, and the only one I can get to come up in the browser. The solution here is just to hit the “Script” button, and modify the generated code to point at the right table, as shown below:

USE[msdb]

GO

CREATE DATABASE AUDIT SPECIFICATION [SQL Agent Audit]
FOR SERVER AUDIT [My Sample Audit]
ADD (UPDATE ON OBJECT::[dbo].[sysjobs] BY [public])

GO

This code runs just fine, and as you will see has the desired effect. As with SQL Server Audits, Database Audit Specifications are created in a disabled state. As before, just right click to enable and the Auditing will begin.

Finally to test, just change the Enabled state of any SQL Agent job on your server, then check the Application Log, where you will find an Event detailing who just changed its status!

SELECT    s.[Name] as [Schema]
,        t.[name] as [Table]
,        SUM(p.rows) as [RowCount]

FROM sys.schemas s
LEFT JOIN sys.tables t
ON s.schema_id = t.schema_id

LEFT JOIN sys.partitions p
ON t.object_id = p.object_id

LEFT JOIN  sys.allocation_units a
ON  p.partition_id = a.container_id

WHERE    p.index_id  in(0,1) -- 0 heap table , 1 table with clustered index
AND        p.rows is not null
AND        a.type = 1  -- row-data only , not LOB

GROUP BY s.[Name], t.[name]
ORDER BY 1,2

For a very swift explanation – sys.schemas and sys.tables list the schemas and tables in the database, so joining these together on schema_id gives a list of all tables by schema in the database. Adding on sys.partitions then pulls in the partitions associated with each table, and finally sys.allocation_units pulls in the allocation units, which i’m not quite sure what they are – the guts of this query were pulled from another blog which I embarrasingly can’t trace back to now.

I’m no expert on DMV’s so if you have any views on the quality of this query – please leave a comment with your thoughts.

More details can be found at the SQL 2008 Official site. The rumour mill seems to imply that SQL 2008 R2 will drop at roughy the same time as Windows 7.

Just in case you have no idea what Master Data Management entails, it covers managing and centralising metadata within an organisation that may span many systems. The most common example is central management of customer records. For more insight and a more detailed explanation, I suggest you read this white paper on the subject by Roger Wolter and Kirk Haselden.

Anyway, onto the webcast – about the 1st quarter of the webcast is the usual generic roadmap blurb, but then the presenter gets into the real meat of Gemini – an Excel based ‘in memory’ analysis tool that allows joining between entities without having to know about such things, superfast analytics – pivoting, calculation, charts etc. and then being able to publish to Sharepoint. From an OLAP analysis point of view, the Pivot Tables also has slicers (effctively table wide filters) displayed in the spreadsheet as well, and it would be good if that made it into Pivot Tables generally in the next release of Office. It looks like an incredible tool and very easy to use – and may be a powerful step towards the realisation of the ‘BI for the masses’ vision. The presenter did let slip one weakness though – much has been made of the 100 million rows of data demo – but that data still has to be loaded into memory first and will still take significant time. I also suspect that how successful Gemini will be is going to depend on how much it will rely on good data structures being in place in an organisation to support it. The Data Warehouse is going to remain the core part of  any succesful BI delivery.

The next component of interest was the reusable Reporting Services components – there is the concept of a library of components that can be built – e.g. standard charts, logos, gauges etc – and then dropped into any report, either by a developer or a user in Report Builder 2.0.  What really grabbed my attention is that these components are version aware – i.e. if the library version of the component is updated then if you reopen the report in design mode it will let you know and give you the option to update. Again this points to ‘BI for the masses’ as you can have developers create some great components which any user can then drop in to their home grown reports. Plus as any developer knows, there’s a lot of repetition and any options for code re-use are always appreciated.

Finally, Project Madison was covered – and seems more about scalability up to multi-terabyte warehouses. It was a bit infrastructure focused for me so most of it passed me by, but clearly Microsoft are stepping up to try and address the market perception that they can’t scale.

Anyway, this all will be dropped in late 2010 as Kilimanjaro – an interim release of SQL Server.

I’ve also added links in the Relevant Blogs section for the SSIS Team Blog and for Jamie Thompson’s excellent SSIS Junkie, where I have found more solutions to SSIS problems that anywhere else on the net.

A clean install of PerformancePoint Server 2007 (PPS) against SQL Server 2008 (SQL2K8) is not directly possible, even at PPS Service Pack 2 (SP2). There is an approach documented on TechNet which involves installing some SQL2K5 components (See scenario 3 of this article: http://technet.microsoft.com/en-us/library/cc514367.aspx) but I couldn’t get it to play ball. I would guess that PPS2007 won’t properly integrate with SQL2K8 for a while yet, so if you are installing PPS2007, do it against SQL2K5 SP2.

Next up, SQL2K5 won’t install cleanly on a vanilla install of Windows Server 2008 (Win2K8). You need to enable the Web Server Role, and then enable features as described in this KnowledgeBase article (http://support.microsoft.com/kb/938245/) – it’s not complicated. It also throws up a warning to patch SQL2K5 to SP2 when running the installer, but I would advise you did that on any install anyway.

Installations are never easy!