So as part of this new database development project I’ve been working on, I’m having to normalize and unwind an unwieldly Excel sheet that’s chock full of intermixed data within the same columns. Values, ranges of values, integers, currency, percentages, and text. Yeah.

As part of the conversion process and figuring out what algorithms to write to unwind the data, I’m using the Visual Studio 2005 Business Intelligence Development Studio to run my data conversion and SQL population tasks. One of the things that’s SERIOUSLY irritated me from the get-go about this tool is that the Excel OleDBConnection object flat out ignores data, or gives its own interpretation as to what it thinks a certain column’s datatype is. (STUPID.) This ended up throwing errors that wouldn’t even output to the specified errorOutput columns – just to a dialog box that said it gave up. In short, it wasn’t able to convert from the datetimestamp datatype from Excel to the datetime datatype in SQL.

I showed this to my resident data warehouse SQL guru guy, who was having a similar issue with a project he was working on. Apparently, there’s a not-so-well-documented “feature” in the OLEDB connection string (used in BIDS) whereby if you append the value IMEX=1 to the end of it, it forces the data flow task to read all intermixed data as a string. Dates, integers, strings, whatever. (Verified by connectionstrings.com). The Data Conversion step in my workflow task FINALLY started behaving normally and doing what I asked it to to – including throwing data into the correct external columns and everything in my test table. WOOHOO!

Now the fun of trying to figure out how to un-wind the cluster of integer strings and values in my dataset. *Sigh* Any cursor gurus out there?