The Haunted (data ware)House of Horrors:
A nightmare journey through warehouse project hell
ESTIMATED READING TIME: 7 MINUTES
As we approach the haunting week of Halloween, images of spooky, evil creatures begin to appear everywhere, from google search pages to the neighbour’s front yards.
But if you’ve spent any amount of time working in organisations that collect large amounts of data that gets funnelled into a corporate dashboard, we can almost guarantee the scariest scene you most often face is the cursed reports generated from bone-chilling and deathly data warehouses.
It’s as if, somewhere in our midst, there lives a suite of devilish fiends, who are dedicated to preventing the courageous, Frodo-like characters brave enough to take on a warehouse project.
Let’s take a quick walk through the harrowing halls of the data warehouse of horrors…
The Wicked Data Warehouse Witch’s Creepy Communication Curse
The Data Warehouse Witch is the first evil character you’ll meet along the road to warehouse hell. She’s a cackling crone who casts her communication curse across all the parties involved in the project.
The creepy communication curse occurs when the business area requesting the build uses the same language as the BA and IT team, and unbeknownst to any of them, they’re meaning completely different things.
The result is a warehouse end product that resembles a cauldron of ingredients no one can decipher and which produce putrid outputs no one wants to use.
The Beastly Blindness Bogeyman
Once you’ve got your communication clear, and the vision for what the build needs to deliver is defined, the next fearsome foe in the warehouse build is the beastly blindness bogeyman.
This guy is a trickster of the highest order, hiding in the shadows of complex data flows.
Our brave warehouse builders begin their model building (which of course, they’re smart enough to know they should do) and yet, despite this, the warehouse they get at the end isn’t functional for most of the people, most of the time.
In fact, that Bogeyman has so effectively blinded our build team that they didn’t realise they didn’t have the full picture until it was too late to change the build.
This curses the warehouse to a life time of breakdowns, cost overruns, and inabilities.
The Zero Accountability Zombie
This dreadful fellow can strike at any time but most often appears during the build when the coders dive head-long into the Zero Accountability Zombie’s clutches. The Zombie refuses to let the coder sense that there is something that could and should be done when data in the fields being matched doesn’t look like it matches.
This kind of irresponsibility builds countless black holes of data cleaning and issue resolving into any warehouse build.
The Ghosts at the Gates of Warehouse Growth
These guys are hard to pin down – they are not made of matter, after all. But boy, they’ll haunt a warehouse for the rest of its days.
When the communication curse, the blindness bogeyman and the accountability zombie have all done their work successfully, the ghosts at the warehouse gate keep the possibility of growing the warehouse to accommodate more business requests as tormented and impossible as fixing the initial problems has been.
How a Fearless IT Frodo can take on the Mount Doom of Warehouse Builds
With all these terrifying figures lining the path to a successful warehouse build, it’s hard to imagine anyone agreeing to take it on. But in the quiet Shire of IT, many a Frodo arise to carry the burden into the fiery centre of Warehouse Mount Doom.
Because we see ourselves as the guys making up the Fellowship of the Ring, helping our Frodo-like clients to achieve their goals, we have written out the easy step-by-step guide to navigating the journey of your next warehouse build.
1. Recognise whether a warehouse is really a good idea or just a cool-sounding dream that isn’t what’s needed. This means really researching and effectively costing the project build, consulting with business areas to determine need vs potential use vs cost/time/effort.
2. If a warehouse is the right decision, (or even if it isn’t, quite frankly this will be a useful activity), form a business term glossary team and get that glossary populated with a comprehensive range of business terms. Write clear, simple and succinct definitions and have the glossary endorsed at the highest level of the organisation as the single source of reference for all communication within the organisation, from this point forward. This ensures clarity in steps 3 and 4.
3. Conduct as much consultation as you can with as many business areas as you can. Knowledge and understanding are the keys to success here. We know this is not always easy and sometimes not possible, but the greater your understanding of the data and information landscape across the entire enterprise, the more solid the foundation you’ll build for your warehouse. And that means greater chance of success for this build, and ease of success for future builds.
4. Transfer this knowledge into a comprehensive model of the organisational information landscape, including business function diagrams, data flow diagrams and, at the least, a very clear warehouse model that will deliver the desired end points and priorities. [If this sounds like a far more torturous ordeal than a haunted data warehouse of horrors could ever be… see our extra note about models and mapping, to the right].
An important part of getting to a successful model is setting up communication channels across the organisation and learning about data and information issues from as many people as possible. This can be a long and large undertaking and it can often feel overwhelming. It will seem to consume time and energy that will feel like it could be better used actually getting on with the building. But take heart – the hero’s journey is always full of tests and trials. Don’t get distracted and start building out of impatience or frustration. Stay the course, plan hard, model well and you’ll avoid the enemies, the blindness bogeyman and the ghosts of locked warehouse gates. Always make sure you know what your end points are and what your path forward will be before you begin to build. (ps your glossary build will have helped you immensely with this process and is necessary for modelling success).
5. Find the data warehouse dashboard software that best suits your data, your end points and your budget. Don’t get me started on the amount of wasted money that gets spent buying warehouse software solutions before the full scope of the model is known and the requirements of the software are articulated. That’s just crazy. And don’t get bogged down in arguments about the architecture – if you truly understand the business strategy, use the architecture and software that can best service that strategy.
6. Buy the right software and begin the build. Please, for the love of all that’s Hallowed, shop around and do your research. Flashy sales people will tell you anything you want to hear. Big companies can create lovely marketing materials. Bells and whistles can appear exciting. But your choice needs to deliver on your end points. Focus on those.
7. Beta test the build and tweak to perfect it. If you’ve followed the plan here, this should be a fairly painless process. If not, well, you’ve just entered the warehouse of horrors, where you can check out anytime you like, but you can never leave.
8. Launch the warehouse. Hear the people cheering because it’s a raging success.
A note about models and mapping
Models and diagrams are often not seen as necessary, but here’s why we’re very insistent on them. Data warehouses are usually built to meet a small scope, but IT are smart enough to recognise that they could help the wider organisation by setting up a bigger delivery platform than the initial scope requires. It’s a great idea, but it fails largely because of lack of modelling. A comprehensive model allows you to build really solid foundations, supporting an initial warehouse that has doors built into it in logical and obvious places. These doors may stay locked and unused for now, but if/when other business areas come and ask for an extension build, those doors can be opened easily and the extensions run smoothly. If you do not do your modelling, those doors are not built in. Walls are built. When the extension requests come in, IT realise they need to knock holes in walls or carry data out the existing doors and back in the front doors of new builds, which costs a LOT more, creates immense potential for failure and simply makes using the warehouse harder.
9. Measure and record the successes This not only makes you look good to your superiors, it is actually helpful for future projects, for justifying the spend on expansion projects and it’s good for ensuring you’ve learned from any mistakes made.
10. Repeat steps four through nine as new extension requests are received. By now, you’re a seasoned fighter of warehouse evil, so go get ‘em!
Have you had experiences in a Haunted (data ware)House of Horrors?
We’d love to hear your spooky ghost story, so please share it over on LinkedIn!
Terry Smith
Terry is a co-founder & Chief Education Officer at Intraversed. She spends her days helping information governance teams implement the Intralign Ecosystem, an award winning methodology that builds stable information foundations for reliable reporting.
Connect on LinkedIn