So, here are my takeaways from day 1:

From the keynote:

Denali becomes SQL2012 and is slated for release in the first half of next year (way to allow yourself some leeway on the final release date!)

Crescent becomes Power View – and will work on multiple mobile platforms – Apple & Android via Browser, WP7 via a richer app

HADOOP is going to be supported in Windows server – the first CTP is due next year. If you have access to the PASS DVD’s / Sessions – see Dr Dewitt’s presentation on HADOOP – very enlightening

From wandering around the summit:

As per Elad Ziklik of the DQS team, performance is one of their focuses for the next release

I got to finally meet Ivan Peev of CozyRoc after many years of email and phone conversations, and he told me about their new SAS Connector – which allows reading and writing to SAS datasets without an actual SAS install.

I also got to meet Matt Masson, guru of the SSIS team who told me about Project Barcelona – a tool that will do data lineage, metadata management and impact analysis via a crawler as opposed to a manually maintained set.

I also got to sit in on a customer feedback session about BI in the Cloud – unfortunately all under NDA – but it was a great forum to discuss and help direct Microsoft’s Cloud BI ambitions.

I also had a chat with fellow Aussie BI guy Roger Noble who told me about a use for the Term Extraction transformation in SSIS – using it to scan through documents and auto-tag them as the were uploaded to SharePoint – which is pretty cool.

So, that was Day 1… Day 2 to follow!

The most important thing to take away is that a Columnstore Index should actually cover the entire table. Its name is a little misleading – the feature is less of an index, and more of a shadow copy of the table’s data, compressed with the Vertipaq voodoo. I suspect they have used the term index because the Columnstore doesn’t cover all data types – the important ones are there, but some extreme decimals and blobs are excluded – for a full list see the MSDN documentation. So for any big table, whack a Columnstore index across the entire table.

Next up is to understand how to use them and how to detect when they are or are not being used. The key thing is to only use them in isolation (e.g. summary queries) or for Inner Joins. Outer Joins don’t work right now, though there are cunning workarounds that apply if you are Outer Joining to summary data – see Eric Hanson’s video referenced below somewhere around the 50 minute mark.

You can detect when they are being used by the Execution Mode described in the Query Plan. This is new in Denali and is either Row or Batch. Row means traditional SQL Server execution and Batch means the Columnstore is being used.

So, the key takeaways:

• For any large table put a Columnstore index across the entire table
• Only join using Inner Joins
• Spot the use of the Columnstore in Query plans via the Execution Mode of Batch

Useful reference material:

• Eric Hanson’s Tech-Ed video (a bit dull, but informative – if you know your Data Warehousing theory, skip the first 15 minutes)
• The TechNet Columnstore Index FAQ
• Columnstore Indexes for Fast Data Warehouse Query Processing in SQL Server 11.0 – Whitepaper by Eric Hanson
• MSDN documentation

What does this mean?

1. First up this only applies to SQL Server connections – other OLE DB connectivity such as Teradata and Oracle will be unaffected.
2. In the short term, nothing – if you are using OLE DB it will carry on working for a fair few years yet – but if you are planning for a systems longevity – look to using ODBC for all your SQL Server connectivity needs

It’s a slightly confusing move given that ODBC underperforms OLE DB – so it’s bad news for us SSIS people, unless part of the change also includes some significant performance and capability improvements.

The tl/dr version is this:

Columnstore indexes use the Vertipaq compression engine (that’s the shiny compression engine in PowerPivot) to further compact indexes to make querying them between 10-100 times faster.

The most significant limitation is that tables become read-only when they have a columnstore index (no Inserts / Updates / Deletes etc) – though James notes you can work around this by using Partitions if you are dealing with tables that are just additive. Otherwise indexes will need to be dropped and recreated as data changes.

So – a powerful new indexing feature which, with careful management – can have a serious positive impact on the performance of your Data Warehouse.

However, where the Microsoft BI stack has a competitor beating edge is the 3rd Party ecosystem – so where there is a gap in the toolset, often another company will step in and fill it. In this case, Data Lineage issues are addressed by a tool created by Pragmatic Works (which is run by Brian Knight, an SSIS heavyweight) called BI Documenter.

BI Documenter Review

So what does the tool do? It has 3 key functions:

Documentation Generation: The tool auto-generates documentation for Databases, SSIS packages, SSAS Cubes and SSRS reports. It’s quick, and the output is pretty – and it’s really a bit useless. Its benefit is if you have to say you’ve produced some documentation and need to do so with minimal effort. The reason I say this is because it provides no context for why things have been done, what the purpose of the component is, where it fits in to the framework etc. My usual gripe with documentation I come across is that it only tells me the what, not the why, and the why helps me solve a problem. This tool can’t do anything to address this.

Data Lineage: Now this is where the main value lies. Through simple navigation you can select any object (table / view / package etc) in the solution and see what objects depend on it and what objects it depends on in turn. This is great in a complex system where if you need to make a change and need to find out what that impacts.

Now its not perfect – it seems to skip documenting some sources, such as flat files, so they get missed in the impact analysis. And the level of granularity is at the object level – for example you can’t see the impact of an individual column change, just at the table level, but its still a great start and a useful tool.

Snapshot comparison: A final piece of value which can be useful in troubleshooting. BI Documenter takes snapshots of your solution to document – and you can compare these to identify changes in the solution. The detail level is pretty good and will be a great place to start tracing changes in your system when your source control systems fail.

Conclusions

Is the tool worth it? At a maximum cost of US$500 a seat, it’s definitely got a place somewhere in your organisation. The documentation tool is of limited use but the Data Lineage and Snapshot comparisons are worth the cost of the product. Full details here: Pragmatic Works – BI Documenter

The design choices in question are around the managing of history data. In one component of the system it relates to Database storage design, the other relates to Cube storage design. In both cases the history data is stored in a separate location to the “current” data.

Databases: Why separate history tables are a bad idea

The first – and most compelling – reason for not storing your history data in separate tables to your current tables is that it increases complexity for users. Instead of having one location to look for data, your users now have to use two.

The second compelling reason is that there is no point to doing this from a storage point of view. SQL 2005 & 2008 (Enterprise editions only, admittedly) provide partitioning. This enables the contents of an individual table to be stored in separate locations on different filegroups. This means that you can store your current days data in one location and your history in a different one. The reason for doing this is the same as splitting it into separate tables – that querying the current section will be faster than the historic section.  In theory queries against partitioned tables should in fact be faster as the current data is now no longer in the same filegroup as the history data.

Now, there is an overhead associated with designing and maintaining partitions but I don’t see that it is significantly larger than that required to deal with the process required to archive data into separate tables on a daily basis. Additionally when maintaining separate history tables, you need to separate out every single table, whether it gets 10 rows a day or 10 million. With partitioning you can just target the large tables that need that focus.

There are other downsides to maintaining separate tables. If you make a change to a table design, you need to do it in 2 places.  You also need to remember to update your history processes. If your history process fails, you can end up with users getting unexpected query results or ETL process failures when the system loads the next day’s data into the current tables –  and untangling it becomes a real mess. If your partition processes fail to run, you just have too much data in one filegroup for a while – unlikely to be fatal.

So if you have large tables you need to split out for performance purposes – do it at the back end, using the power of the database – which is designed to store data efficiently. Keep it away from the users – they neither need to know or care about your need to keep the data separate. If you want to give them a single object to query with the current day’s data, just use views.

Cubes: Why a separate history Cube is a bad idea

Much of the above applies here – SSAS also has partitions – so you can again store your historic and current data in separate physical locations with the users being totally unaware of this. Again there is overhead in maintenance, but this will also balance out with the maintenance and risks associated with maintaining two identical cubes that only differ in terms of data source.

Use your storage options!

So without banging on about the same things any further, please consider the following two points whenever considering managing your history data:

1. How does what i’m planning affect my users?
2. How does what i’m planning leverage the platforms capabilities?

An Alias is an Operating System level setting that allows you give a friendly name to your Server – and you can then connect to the server using that friendly name. What this means is that as you migrate your code from environment to environment, if you use the same friendly name for your SQL Server in each tier of the deployment, you don’t have to change your connection strings.

It’s a neat trick (though i’m not sure how the names resolve if you use the same name in environments that can see each other) – it certainly requires some proper planning from an infrastructure point of view.

Lunchtime Wed 3rd March, 2010 – Ensuring Optimal Performance in SQL Server 2008 Based Applications with Viktor Isakov

Evening * Thu * 4th Mar 2010 – What’s new in Reporting Services 2008 R2 and PerformancePoint Services 2010 with Peter Myers, presenting what’s new in the upcoming release of Reporting Services 2008 R2 and PerformancePoint Services 2010

And TBA date in April 2010 – Knights of the SSIS Round Table – Kevin Wong, Glyn Llewelyn and myself will be presenting a series of mini demos followed by Q&A, so a chance to pick some expert brains

Hope to catch you at one of the sessions!

Enter SQL Server Auditing, introduced in 2008.

What is SQL 2008 Auditing?

Well, you could read the official documentation on MSDN, but it’s a little overwhelming, so here’s the view from 500 ft. SQL Server Audit provides a secure means of tracking access and changes made to the database schema or its data. What this means is you can trace who tried to access what and when. If any command was issued to change the data, you can track this as well. You cannot track the changes made to the data. So for example if someone made an insert into a table, you could track that someone made an insert, who did it, when that insert was made and what the SQL of the insert statement looked like – except for what data was inserted. That task falls to Change Data capture (CDC) which isn’t directly tied to Auditing (currently).

How do I implement Auditing?

A SQL Server Audit is made up of two components, the first of which is the SQL Server Audit. This is a server level object (created in the master database) that defines where logs will be written. Logs can be written to either a file, the Application Log or Security Log. These must be created before you create any actual Audits.

The second component is either a Server Audit or Database Audit. Server Audits again exist at the server level, and track activities that occur at the server level, such as login attempts or permission changes. Database Audits exist within an individual database and track activities at a database level such as schema changes or data operations.

You can have multiple SQL Server Audits defining multiple log target locations. Multiple Server and Database audits can be created to use a given SQL Server Audit. This probably makes more sense when explained in pictures:

Fig 1: SQL 2008 Auditing Hierarchy

So how do I audit my SQL Agent Jobs?

Setting up Audits can either be done through SQL Server Management Studio (SSMS) or through SQL scripts. As usual any action carried out through the Wizard, so I will do the wizard first, then show the generated scripts.

Step 1 is setting up the SQL Server Audit. To create this in SSMS, under the Security Node of the Server is a folder called “Audits”. To create a new Audit, simply right click on the folder and choose “New Audit..”, as set out below:

Fig 2: Creating a SQL Server Audit

Then in the Wizard just enter the Audit name and in the dropdown select “Application Log”, as shown below. The alternatives are to the Security Log or to a File, but I won’t be covering those in this simple example. Click on OK and you’re done.

Fig 3: Creating a SQL Server Audit

The equivalent SQL script is below:

USE [master]

GO

CREATE SERVER AUDIT [My Sample Audit]
TO APPLICATION_LOG
WITH
( QUEUE_DELAY = 1000
 ,ON_FAILURE = CONTINUE
)

GO

Now you have somewhere to write your log. Note that by default SQL Server Audits are disabled when they are created, which means nothing will be written to your log until until it is enabled. This can be done just by right clicking on it in SSMS and choosing “Enable”.

The next bit is to set up the Database Audit on the SQL Agent Jobs table, which is in the system table [msdb].[dbo].[sysjobs]. Under the msdb database, open up the “Security” folder and within there is the “Database Audit Specifications” folder. Right click here and choose “New Database Audit Specification…” as shown below.

Fig 4: Creating a Database Audit Specification

This will open the Create Database Audit Specification dialog. Because we are operating on msdb, unless you are logged in as sa, you probably won’t be able to create objects and the process will fail. So ensure the user you are logged in as has appropriate permissions on msdb. Because I’m playing on a Dev environment, I just allowed my own user access to msdb and gave them db_owner rights – this probably won’t fly in a production environment, of course!

The dialog is shown below. First task is to choose which SQL Server Audit to write to – so we use the one we created above called ”My Sample Audit”. You choose the Audit Action Type – i.e. what you want to log. In our case we want to see when changes are made to our jobs in SQL Agent, so we choose the “UPDATE” Action Type. Next we have the Object Class, which for this case is a Database Object, so “Object” is selected, and then the Object Name. Finally the Principal defines who is Audited – in the example I have selected “public” because every user is in the public role, so I will capture any users UPDATE commands issued on my selected table.

Fig 5: Creating a Database Audit Specification

Now here we hit a minor hiccup – you will note in the example above I have selected the object sysdtslog90 – it’s the only non-system view in the database, and the only one I can get to come up in the browser. The solution here is just to hit the “Script” button, and modify the generated code to point at the right table, as shown below:

USE[msdb]

GO

CREATE DATABASE AUDIT SPECIFICATION [SQL Agent Audit]
FOR SERVER AUDIT [My Sample Audit]
ADD (UPDATE ON OBJECT::[dbo].[sysjobs] BY [public])

GO

This code runs just fine, and as you will see has the desired effect. As with SQL Server Audits, Database Audit Specifications are created in a disabled state. As before, just right click to enable and the Auditing will begin.

Finally to test, just change the Enabled state of any SQL Agent job on your server, then check the Application Log, where you will find an Event detailing who just changed its status!

SELECT    s.[Name] as [Schema]
,        t.[name] as [Table]
,        SUM(p.rows) as [RowCount]

FROM sys.schemas s
LEFT JOIN sys.tables t
ON s.schema_id = t.schema_id

LEFT JOIN sys.partitions p
ON t.object_id = p.object_id

LEFT JOIN  sys.allocation_units a
ON  p.partition_id = a.container_id

WHERE    p.index_id  in(0,1) -- 0 heap table , 1 table with clustered index
AND        p.rows is not null
AND        a.type = 1  -- row-data only , not LOB

GROUP BY s.[Name], t.[name]
ORDER BY 1,2

For a very swift explanation – sys.schemas and sys.tables list the schemas and tables in the database, so joining these together on schema_id gives a list of all tables by schema in the database. Adding on sys.partitions then pulls in the partitions associated with each table, and finally sys.allocation_units pulls in the allocation units, which i’m not quite sure what they are – the guts of this query were pulled from another blog which I embarrasingly can’t trace back to now.

I’m no expert on DMV’s so if you have any views on the quality of this query – please leave a comment with your thoughts.