<?xml version="1.0" encoding="ISO-8859-1"?>

<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/">
	<channel>
		<title>CORTEX Forums - BI Monkey</title>
		<link>http://www.tbig.com.au/forums/</link>
		<description>BI Monkey is the ‘nom de plume’ of James Beresford, a Certified Microsoft BI Professional and MBA living and consulting in Sydney</description>
		<language>en</language>
		<lastBuildDate>Thu, 09 Sep 2010 05:01:51 GMT</lastBuildDate>
		<generator>vBulletin</generator>
		<ttl>60</ttl>
		<image>
			<url>http://www.tbig.com.au/forums/images/styles/tbig/misc/rss.jpg</url>
			<title>CORTEX Forums - BI Monkey</title>
			<link>http://www.tbig.com.au/forums/</link>
		</image>
		<item>
			<title>Convert Text Stream to String</title>
			<link>http://www.tbig.com.au/forums/bi-monkey/15685-convert-text-stream-string.html</link>
			<pubDate>Mon, 06 Sep 2010 01:22:27 GMT</pubDate>
			<description>One of the ongoing challenges with SSIS is its difficulty in handling complex or damaged text files. One approach to dealing with such files is to...</description>
			<content:encoded><![CDATA[<div>One of the ongoing challenges with SSIS is its difficulty in handling complex or damaged text files. One approach to dealing with such files is to <a href="http://www.bimonkey.com/2009/06/flat-file-source-error-the-column-delimiter-for-column-columnname-was-not-found/" target="_blank">bring them all in as one*wide text column and then split them using code</a>. Sometimes, the file is too wide for that approach, so below is an extension of that method where you import the column as a text stream (DT_TEXT, or Unicode DT_NTEXT) and then split the text stream in a script transformation:<br />
<blockquote>******* <font color="#008000">&#8216; Declare variables</font><br />
******* <font color="#0000ff">Dim </font>TextStream <font color="#0000ff">As</font> <font color="#0000ff">Byte</font>()*********** <font color="#008000">&#8216; To hold Text Stream</font><br />
******* <font color="#0000ff">Dim</font> TextStreamAsString <font color="#0000ff">As String</font>**<font color="#008000">* &#8216; To Hold Text Stream converted to String</font><br />
******* <font color="#0000ff">Dim</font> StringArray() <font color="#0000ff">As String</font>*******<font color="#008000">* &#8216; To contain split Text Stream</font><br />
<br />
<b>*******<font color="#008000"> &#8216; Load Text Stream into variable</font><br />
******* TextStream = Row.TextStreamColumn.GetBlobData(0, <font color="#0000ff">CInt</font>(Row.Column0.Length))</b><br />
<br />
<b>*****<font color="#008000">** &#8216; Convert Text Stream to string</font><br />
******* TextStreamAsString = System.Text.Encoding.ASCII.GetString(TextStream)</b><br />
<br />
******* <font color="#008000">&#8216; Split string into array and output</font><br />
******* StringArray = TextStreamAsString.Split(<font color="#800000">&#8220;#&#8221;</font>) *******<br />
<br />
******* Row.Column1 = StringArray(1).ToString<br />
******* Row.Column2 = StringArray(2).ToString<br />
******* Row.Column3 = StringArray(3).ToString**<br />
<br />
</blockquote>An important thing to note*is that in the step where the Text Stream is converted to a string, the Encoding will depend on the type of text stream you are bringing in &#8211; Unicode files will need*&#8221;Unicode&#8221; instead of &#8220;ASCII&#8221;. Also I have used a hash (&#8221;#&#8221;) as the*column delimiter but that value will vary depending on what type of file you are bringing in.<br />
<br />
<br />
<br />
<a href="http://www.bimonkey.com/2010/09/convert-text-stream-to-string/" target="_blank">Get More from the original blog...</a></div>

]]></content:encoded>
			<category domain="http://www.tbig.com.au/forums/bi-monkey/">BI Monkey</category>
			<dc:creator>James Beresford</dc:creator>
			<guid isPermaLink="true">http://www.tbig.com.au/forums/bi-monkey/15685-convert-text-stream-string.html</guid>
		</item>
		<item>
			<title>Modifying an SSIS Package through code</title>
			<link>http://www.tbig.com.au/forums/bi-monkey/15636-modifying-ssis-package-through-code.html</link>
			<pubDate>Thu, 02 Sep 2010 12:29:11 GMT</pubDate>
			<description>Part of any SSIS development experience inevitably results in you discovering a minor mistake or something that was missed a long way into the...</description>
			<content:encoded><![CDATA[<div>Part of any SSIS development experience inevitably results in you discovering a minor mistake or something that was missed a long way into the development cycle &#8211; or even after, in testing (you do test your code, right?). Then you are faced with the tedious job of opening every single package, making a change in every one&#8230; and getting some serious mouse finger. Much like I once did when I learned about <a href="http://www.bimonkey.com/2008/04/blobtempstoragepath-and-buffertempstoragepath/" target="_blank">BufferTempStoragePath</a>.<br />
<br />
Fortunately, there is a way to automate these fixes. The SSIS Object model is (relatively) easily manipulated through .NET languages &#8211; so it&#8217;s not too difficult to write a small program that will change your package. Below is a sample I knocked up that will add a variable to an existing package and save the change:<br />
<blockquote><font color="#0000ff">using </font>System;<br />
<font color="#0000ff">using</font>Microsoft.SqlServer.Server;<br />
<font color="#0000ff">using</font>Microsoft.SqlServer.Dts.Runtime;<br />
<br />
<font color="#0000ff">namespace</font> Package_Modifier<br />
{<br />
*** <font color="#0000ff">class</font> Program<br />
*** {<br />
******* <font color="#0000ff">static void</font> Main(<font color="#0000ff">string</font>[] args)<br />
******* {<br />
*********** <font color="#008000">// Initialize an Application and Package object</font><br />
*********** <font color="#33cccc">Application </font>app = <font color="#0000ff">new</font> <font color="#33cccc">Application</font>();<br />
*********** <font color="#33cccc">Package</font> package = <font color="#0000ff">null</font>;<br />
<br />
*********** <font color="#008000">// Set a package path</font><br />
*********** <font color="#33cccc">String</font>pkgPath = <font color="#800000">&#8220;C:\\<acronym title="Business Intelligence">BI</acronym> Monkey\\SamplePackage.dtsx&#8221;</font>;<br />
<br />
*********** <font color="#008000">// Load the package in package object</font><br />
*********** package = app.LoadPackage(pkgPath, <font color="#0000ff">null</font>);<br />
<br />
<b>*********** <font color="#008000">// Add the new variable</font><br />
*********** package.Variables.Add(<font color="#800000">&#8220;NewVar&#8221;</font>, <font color="#0000ff">false</font>, <font color="#800000">&#8220;User&#8221;</font>, 0);</b><br />
<br />
*********** <font color="#008000">// Save the package</font><br />
*********** app.SaveToXml(pkgPath, package, <font color="#0000ff">null</font>);<br />
********* }<br />
*** }<br />
}<br />
<br />
</blockquote>You can essentially make any change you like to a package &#8211; I&#8217;ve chosen adding a variable because it&#8217;s an*easy manipulation of the*package object*and I&#8217;ve got a long way to go before I work out how to do anything much harder <img src="http://www.bimonkey.com/wp-includes/images/smilies/icon_smile.gif" border="0" alt="" /> <br />
<br />
<br />
<br />
<a href="http://www.bimonkey.com/2010/09/modifying-an-ssis-package-through-code/" target="_blank">Get More from the original blog...</a></div>

]]></content:encoded>
			<category domain="http://www.tbig.com.au/forums/bi-monkey/">BI Monkey</category>
			<dc:creator>James Beresford</dc:creator>
			<guid isPermaLink="true">http://www.tbig.com.au/forums/bi-monkey/15636-modifying-ssis-package-through-code.html</guid>
		</item>
		<item>
			<title>SSIS ETL Framework v1 goes Beta!</title>
			<link>http://www.tbig.com.au/forums/bi-monkey/15084-ssis-etl-framework-v1-goes-beta.html</link>
			<pubDate>Mon, 09 Aug 2010 02:53:12 GMT</pubDate>
			<description>A quick update on the status of the BI Monkey SSIS ETL Framework (on Codeplex @ http://ssisetlframework.codeplex.com/) 
 
The Framework v1 has gone...</description>
			<content:encoded><![CDATA[<div>A quick update on the status of the <acronym title="Business Intelligence">BI</acronym> Monkey SSIS <acronym title="Data Integration">ETL</acronym> Framework (on Codeplex @ <a href="http://ssisetlframework.codeplex.com/" target="_blank">http://ssisetlframework.codeplex.com/</a>)<br />
<br />
The Framework v1 has gone into Beta &#8211; i&#8217;ve completed all the testing, and now just need to tidy up the reports and add some extra logging capability &#8211; but it is now fully functional. I&#8217;ve left it in Beta as I want to get some feedback on it before I move it live, plus fix those small details I mentioned.<br />
<br />
So now I will press on with updating the documentation (yes, really!)*and start laying the foundations for the more Enterprise level v2 framework.<br />
<br />
I look forward to your feedback &#8211; please take advantage of Codeplex&#8217;s issue logging functionality to help me manage bugs and improvements.<br />
<br />
<br />
<br />
<a href="http://www.bimonkey.com/2010/08/ssis-etl-framework-v1-goes-beta/" target="_blank">Get More from the original blog...</a></div>

]]></content:encoded>
			<category domain="http://www.tbig.com.au/forums/bi-monkey/">BI Monkey</category>
			<dc:creator>James Beresford</dc:creator>
			<guid isPermaLink="true">http://www.tbig.com.au/forums/bi-monkey/15084-ssis-etl-framework-v1-goes-beta.html</guid>
		</item>
		<item>
			<title>BI Documenter from Pragmatic Works</title>
			<link>http://www.tbig.com.au/forums/bi-monkey/15010-bi-documenter-pragmatic-works.html</link>
			<pubDate>Wed, 04 Aug 2010 11:07:06 GMT</pubDate>
			<description>I recently had to demo SSIS to an enterprise as part of an ETL tool evaluation. One of the Microsoft BI stacks weaknesses is the lack of Data Lineage...</description>
			<content:encoded><![CDATA[<div>I recently had to demo SSIS to an enterprise as part of an <acronym title="Data Integration">ETL</acronym> tool evaluation. One of the Microsoft <acronym title="Business Intelligence">BI</acronym> stacks weaknesses is the lack of Data Lineage tracking. What this means is there is nothing embedded in the toolset that allows you to identify clearly the source of a data item in a package / report / <acronym title="OLAP = Online Analytical Processing">cube</acronym> without digging into the development environment. Rumours are this will be fixed in the next release, however nothing has yet been confirmed.<br />
<br />
However, where the Microsoft <acronym title="Business Intelligence">BI</acronym> stack has a competitor beating edge is the 3rd Party ecosystem &#8211; so where there is a gap in the toolset, often another company will step in and fill it. In this case, Data Lineage issues are addressed by a tool created by <a href="http://pragmaticworks.com/" target="_blank">Pragmatic Works</a> (which is run by Brian Knight, an SSIS heavyweight) called <acronym title="Business Intelligence">BI</acronym> Documenter.<br />
<br />
<b><acronym title="Business Intelligence">BI</acronym> Documenter Review</b><br />
<br />
So what does the tool do? It has 3 key functions:<br />
<br />
<b>Documentation Generation:</b> The tool auto-generates documentation for Databases, SSIS packages, SSAS Cubes and SSRS reports. It&#8217;s quick, and the output is pretty &#8211; and it&#8217;s really a bit useless. Its benefit is if you have to say you&#8217;ve produced some documentation and need to do so with minimal effort. The reason I say this is because it provides no context for why things have been done, what the purpose of the component is, where it fits in to the framework etc. My usual gripe with documentation I come across is that it only tells me the what, not the why, and the why helps me solve a problem. This tool can&#8217;t do anything to address this.<br />
<br />
<b>Data Lineage:</b> Now this is where the main value lies. Through simple navigation you can select any object (table / view / package etc) in the solution and see what objects depend on it and what objects it depends on in turn. This is great in a complex system where if you need to make a change and need to find out what that impacts.<br />
<br />
Now its not perfect &#8211; it seems to skip documenting some sources, such as flat files, so they get missed in the impact analysis. And the level of granularity is at the object level &#8211; for example you can&#8217;t see the impact of an individual column change, just at the table level, but its still a great start and a useful tool.<br />
<br />
<b>Snapshot comparison:</b> A final piece of value which can be useful in troubleshooting. <acronym title="Business Intelligence">BI</acronym> Documenter takes snapshots of your solution to document &#8211; and you can compare these to identify changes in the solution. The detail level is pretty good and will be a great place to start tracing changes in your system when your source control systems fail.<br />
<br />
<b>Conclusions</b><br />
<br />
Is the tool worth it? At a maximum cost of US$500 a seat, it&#8217;s definitely got a place somewhere in your organisation. The documentation tool is of limited use but the Data Lineage and Snapshot comparisons are worth the cost of the product. Full details here: <a href="http://pragmaticworks.com/Products/Business-Intelligence/BIDocumenter/Default.aspx" target="_blank">Pragmatic Works &#8211; BI Documenter</a><br />
<br />
<br />
<br />
<a href="http://www.bimonkey.com/2010/08/bi-documenter-from-pragmatic-works/" target="_blank">Get More from the original blog...</a></div>

]]></content:encoded>
			<category domain="http://www.tbig.com.au/forums/bi-monkey/">BI Monkey</category>
			<dc:creator>James Beresford</dc:creator>
			<guid isPermaLink="true">http://www.tbig.com.au/forums/bi-monkey/15010-bi-documenter-pragmatic-works.html</guid>
		</item>
		<item>
			<title>Passed 70-448: SQL 2008 BI Developer</title>
			<link>http://www.tbig.com.au/forums/bi-monkey/14741-passed-70-448-sql-2008-bi-developer.html</link>
			<pubDate>Thu, 22 Jul 2010 11:56:32 GMT</pubDate>
			<description>Hurrah! At long last I finally got round to taking 70-448: SQL 2008 BI Developer...</description>
			<content:encoded><![CDATA[<div>Hurrah! At long last I finally got round to taking <a href="http://www.microsoft.com/learning/en/us/exam.aspx?ID=70-448&amp;locale=en-us" target="_blank">70-448: SQL 2008 BI Developer</a> and passed!<br />
<br />
 <b>Test Exams</b><br />
<br />
 As well as blowing my own trumpet, there is also a point to this post about studying for these exams &#8211; and specifically using practice exams. This is a bit controversial &#8211; as evidenced by this recent thread in the <a href="http://social.msdn.microsoft.com/Forums/en-US/sqlintegrationservices/thread/57f43a3a-a926-46ee-ad56-aa568a81a9fe?prof=required" target="_blank">MSDN SSIS Forums</a>. Some exams are a brain dump &#8211; like Test King &#8211; and if you just want to learn the questions and answers, then that will serve you well, but your certification will be worthless once people find that you don&#8217;t actually know what you are talking about.<br />
<br />
 I&#8217;ve been using <a href="http://www.transcender.com/" target="_blank">Transcender</a>&#8217;s exams to get me through 70-448 (and it&#8217;s 2005 predecessor) and found in useful because it helped me understand what kind of questions I&#8217;d be facing, but also where I needed to study. So rather than a rote learning exercise, I used them as a springboard to guide where I needed to study.<br />
<br />
 I&#8217;m not going to endorse Transcender particularly &#8211; I found myself having to correct far too manyof their questions where either the question was misleading, or the answer was simply wrong. To their credit they did actually respond to the issues I raised and corrected them, but it&#8217;s frustrating when you review your answers from a practice run and come to the realisation that the exam itself was wrong.<br />
<br />
 I&#8217;ve also found their customer service unacceptably slow &#8211; being a consultant I migrate laptops and o/s a lot and have burnt through my allotted product activations &#8211; I&#8217;ve had to request a manual activation code. I did this 4 days ago and am still waiting&#8230;<br />
<br />
 <b>Also, the Exam Sucks &#8211; a bit</b><br />
<br />
 Some of the critiques levelled above at Transcender&#8217;s practice exam also apply to the real thing. Some of the questions and answers are vaguely worded or use terms that don&#8217;t match the product itself.<br />
<br />
 However my biggest bugbear is the scope of the exam. There&#8217;s quite a few questions on things that bear no relevance to the day to day experience of a <acronym title="Business Intelligence">BI</acronym> Developer. I point heavily to the questions on SSRS Configuration files which in practice are rarely touched, or the emphasis on the use of Checkpoints in SSIS which most people avoid due to their poor implementation. But I guess this is true of many exams regardless of subject so is a bit forgivable.<br />
<br />
 Regardless, I still think certifications have their value &#8211; they force you to study areas you are less familiar with. It won&#8217;t make you proficient, but it will at least make you aware of capabilities. Experience, however, still trumps a bit of paper <img src="http://www.bimonkey.com/wp-includes/images/smilies/icon_smile.gif" border="0" alt="" /> <br />
<br />
 <br />
<br />
<a href="http://www.bimonkey.com/2010/07/passed-70-448-sql-2008-bi-developer/" target="_blank">Get More from the original blog...</a></div>

]]></content:encoded>
			<category domain="http://www.tbig.com.au/forums/bi-monkey/">BI Monkey</category>
			<dc:creator>James Beresford</dc:creator>
			<guid isPermaLink="true">http://www.tbig.com.au/forums/bi-monkey/14741-passed-70-448-sql-2008-bi-developer.html</guid>
		</item>
		<item>
			<title><![CDATA[SSRS & Stored Procedures]]></title>
			<link>http://www.tbig.com.au/forums/bi-monkey/14652-ssrs-stored-procedures.html</link>
			<pubDate>Mon, 19 Jul 2010 02:51:17 GMT</pubDate>
			<description>As an ETL Monkey, my experience with reports has been a bit incidental. One thing that puzzled me though is why report developers always wrote stored...</description>
			<content:encoded><![CDATA[<div>As an <acronym title="Data Integration">ETL</acronym> Monkey, my experience with reports has been a bit incidental. One thing that puzzled me though is why report developers always wrote stored procedures to generate data for their reports, instead of using the SQL capabilities within the report. They said &#8220;Best Practice&#8221; and I was happy to leave them to it!<br />
<br />
<a href="http://jahaines.blogspot.com/2009/11/ssrs-should-i-use-embedded-tsql-or.html" target="_blank">In this detailed post Adam Haines explains why</a>. It&#8217;s very detailed so here are the key points if you have a short attention span:<br />
<ul><li>The query can be maintained independently of SSRS, allowing tuning the query without accessing or modifying the reports</li>
<li>The execution plans can be cached if you use an Stored Procedure, but not if you use SSRS</li>
<li>Stored procedures allow the use of certain objects that cannot be used in embedded T-SQL in the report such as temp tables and indexes specific to those temp tables as well as table variables</li>
<li>Stored Procedures provide a layer of abstraction between the report and the business logic</li>
<li>Stored Procedures allow re-use of similar logic</li>
</ul>Credit for the above list*to my colleague John Simon who authored most of the above points in an internal discussion.<br />
<br />
<br />
<br />
<a href="http://www.bimonkey.com/2010/07/ssrs-stored-procedures/" target="_blank">Get More from the original blog...</a></div>

]]></content:encoded>
			<category domain="http://www.tbig.com.au/forums/bi-monkey/">BI Monkey</category>
			<dc:creator>James Beresford</dc:creator>
			<guid isPermaLink="true">http://www.tbig.com.au/forums/bi-monkey/14652-ssrs-stored-procedures.html</guid>
		</item>
		<item>
			<title>Infrastructure pains</title>
			<link>http://www.tbig.com.au/forums/bi-monkey/14622-infrastructure-pains.html</link>
			<pubDate>Fri, 16 Jul 2010 06:20:57 GMT</pubDate>
			<description>One of the headaches that has plagued various projects I have been part of has been Infrastructure. From the provisioning of environments, to dodgy...</description>
			<content:encoded><![CDATA[<div>One of the headaches that has plagued various projects I have been part of has been Infrastructure. From the provisioning of environments, to dodgy release practices, to environments that were deemed &#8220;unnecessary&#8221; &#8211; sometimes the problems have not been the code, but where and how it gets into the wild.<br />
<br />
<b>Dev -> Test -> Prod</b><br />
<br />
At the very least, any software release should go through these basic code promotion steps. When you&#8217;re doing a <acronym title="Business Intelligence">BI</acronym> project, just because its a bit odd in software terms, doesn&#8217;t mean you can skip the standard code promotion activities. Development should be done in the Dev environment, where any damage done by bad code is minimal. Once it &#8220;works&#8221; it needs to be promoted into a Test environment to ensure that it actually does work, and it&#8217;s not a fluke of the right test data having been constructed. Once its been tested, it can then go into Production.<br />
<br />
This means on a project you need these three environments up and running from the outset. It can sometimes be a hard sell to smaller, less experienced IT departments who haven&#8217;t experienced the pain of an overly keen developer trashing production IT infrastructure. It does increase cost, but prevention is far better than cure.<br />
<br />
As far as how the environments look, Dev can be whatever you like &#8211; the databases can be a mess, the code can be spaghetti &#8211; who cares? This is the developers playground and they can do what they like in here. However Test and Prod should be exactly the same. This way you can spot the &#8220;but it worked in Development&#8221; problems that somehow drag down production.<br />
<br />
<b>Dev = Test = Prod</b><br />
<br />
Now, the next important thing is to ensure each of these environments are physically the same. So all the software is the same, it&#8217;s patched to the same version, it has the same network cards and drive mappings. If you have a seperate database and SSAS box, don&#8217;t just use one machine in Dev and Test because &#8220;in theory&#8221; it&#8217;s the same as production.<br />
<br />
This -*like the code promotion cycle above &#8211; is about prevention being better than cure. I&#8217;ve wasted many hours of my development life debugging issues that eventually turned out to be due to inconsistent patching, drive letters not being the same in different environments and so on. One of the issues with inconsistent environments is that after a while you accommodate for them &#8211; and forget you are doing it &#8211; then a new developer comes on board and blows things up because you&#8217;ve become so accustomed to the workarounds you&#8217;ve almost forgotten they&#8217;re there.<br />
<br />
The key here is to have a good infrastructure build guide that explains how each environment is constructed, so the Infrastructure team have no excuse for building inconsistent environments. There will of course be some differences &#8211; IP addresses, Server Names, etc. &#8211; but these will be documented and should be legitimate.<br />
<br />
<b>Dev -> Test = Dev -> Prod</b><br />
<br />
Finally, code promotion should be the same regardless of environment. If you find yourself making allowances for a quirk in one environment&#8230; well, see my comments above about inconsistent environments. Code promotion should be a boring routine that can be done by anyone who can follow simple instructions. Because in theory, your developers should have no access to production environments and the promotion from Dev to Test should be considered a dry run for the promotion to Prod.<br />
<br />
<b>Surely this is a bit too much?</b><br />
<br />
Yes, yes it is. If we all coded perfectly and when we deployed we never made a mistake it&#8217;s totally unneccessary <img src="http://www.bimonkey.com/wp-includes/images/smilies/icon_smile.gif" border="0" alt="" /> <br />
<br />
<br />
<br />
<a href="http://www.bimonkey.com/2010/07/infrastructure-pains/" target="_blank">Get More from the original blog...</a></div>

]]></content:encoded>
			<category domain="http://www.tbig.com.au/forums/bi-monkey/">BI Monkey</category>
			<dc:creator>James Beresford</dc:creator>
			<guid isPermaLink="true">http://www.tbig.com.au/forums/bi-monkey/14622-infrastructure-pains.html</guid>
		</item>
	</channel>
</rss>
