Our theme will be “solving data headaches” – understand how the data management core principles of Data Governance, Data Quality, MDM and Data Integration can make your life easier!

This post was originally published on the dataqualitymatters blog!

Have a good weekend and  we will see some of you next week.

Compliance generates reams of data that need to be managed, adding  complexity onto an already overwhelming data environment. The reality is that compliance is a burning pain for organisations across all  industries and sectors, with a long and continuously growing list of compliance legislation and guidelines including: Sarbanes Oxley; King III; The Protection of Personal Information Bill (POPI); Basel II and  Basel III; Solvency II; the Consumer Protection Act (CPA); the  Financial Intelligence Centre Act (FICA), the Regulation and Interception of Communications Act (RICA), Legal Entity Identifier  (LEI), Public Finance Management Act (PFMA) and the latest Foreign Account Tax Compliance Act (FATCA).

A report recently issued by Aberdeen Research indicates that almost half of finance employees are “challenged by the fact that their organizations are leveraging risk and compliance data in different formats, making it difficult to compare data.”  According to the report, complying with regulations is a key concern for CFOs. And a distressing number of respondents indicated that the existing IT  infrastructure is lacking in the advanced capabilities needed to support governance, risk and compliance (GRC) initiatives.

This is far from the end of the problem, however. Gartner predicts further headaches, stating that “by 2016, 20% of CIOs in regulated industries will lose their jobs for failing to implement the discipline of information governance successfully”. Gartner also recommends that these regulated businesses invest in information archiving technology in order to bring data under control.

While Business Intelligence (BI) is touted as the answer to help businesses make sense of their data avalanche, the reality could not be further from the truth. BI has become yet another headache for organisations, along with Master Data Management (MDM), Enterprise Resource Planning (ERP), Customer Relationship Management (CRM), Extract Transform Load (ETL) and a host of other tools. Despite the lauded ability of BI to help businesses improve decision making, the majority of BI implementations fail or never get off the ground.

One of the main reasons for this failure is the fact that the underlying data is fragmented, duplicated, inaccurate, irrelevant and outdated. While BI has the potential to deliver fast, reliable information and ‘intelligence’ along with a single version of the truth, this entire house of cards is balanced on one single point of  failure – the quality of the data.

Maintaining accurate data, which in turn facilitates compliance and paves the way for successful BI and MDM, is key in providing the cure for data headaches. However, this typically proves to be a major challenge, mainly due to inefficient processes or reliance on inappropriate technology. Various people may have different ways of documenting the same information and without processes and tools in place to ensure standardisation, this leads to duplication and inaccurate data.

Fortunately there is a remedy at hand. Data governance – a combination of disciplines, improved processes and the right technology. If pragmatically applied data governance should assist business to cut through the overload and identify and address the critical data issues that will drive the biggest returns, resulting in clean data that deliver results and information that is accurate. Quality data delivers insight based on fact rather than guesswork, addressing myriad compliance regulations and is the key to successful BI, MDM, CRM and back office implementations.

Solving the data quality problem will provide a painkiller for the majority of data headaches.

This post was originally published on the dataqualitymatters blog

It’s amazing how fast new data gets captured. Servers are filled, or modified, or copied without any sense of what may be required to get that information out in a usable way. Add on a lack of consistent data entry, or even inconsistent naming conventions and you could find yourself with data that is nearly unusable.

Now try to consolidate information from many systems, with their own quirks and inconsistencies, and the problem grows exponentially. People want to sort through these massive amounts of data to create an accurate master record for client or product data, or to generate reports for spend analysis or for regulatory compliance.

The old children’s game neatly summarises the relationship between IT and Business when discussing data quality and data governance.

Imagine that in the early flushes of your first major love you are forced to move hundred of miles across country, separated from the object of your desire. As your emotions well up you choose to sit and write a passionate ode to your beloved – deeply personal and describing your feelings. A letter intended to woo and win!

Pushing the send button on your mail you expect the message to be delivered – without delivery the passion is in vain. You also expect the letter to get there uncensored (and unchanged).

IT has a critical role to play in any data governance or data quality process. They own the infrastructure, they are responsible for security, they must “deliver the letter.”

But you, the business person, own the letter.  Noone else can possible understand your passion, can describe the way you feel, can successfully woo your love for you. Business users who are intimately familiar with the business use and value of data must take ownership of ensuring that the standards set are correct and in lie with business needs.

IT can only make changes to data if they are in line with these business standards. And critical content should not be changed at all.

Data governance and data quality projects are critically dependent on business knowledge. Equally, a project on any meaningful scale requires technical and system support.  Both business and IT must be involved.

In the ideal world your letters will be compelling and they will be received.

This post was originally published in the dataqualitymatters blog

Predictions by Boris Evelson of Forrester Research suggest that delivery will be a primary focus for BI projects for the coming year, and a number of these are relevant to the data quality community.

Of particular note is the prediction that enterprise standards will not be allowed to stand in the way of delivery; in other words, specialised data quality tools with the functionality to get things done will trump architecture standards.

Leading IT departments are taking cognisance of the fact that the traditional reliance on a few big vendors, who according to Gartner Research are focussed on protecting the status quo rather than on delivering new value, is insufficient to meet the enterprise data quality needs, and are turning to specialist platforms. The integration of an enterprise data quality platform, such as the Trillium Software System®, into business applications and real-time processes is critical to support the increasing demand for near real-time analytics.

Another key prediction is that BI will move into the hands of end users (and IT will have to accept this), but that this self-service element must be controlled and managed at an appropriate level. This continues a trend that has been developing over the last few years in the data quality space, particularly within leading organisations that recognise the importance of business user involvement in data quality and data governance projects, as it enables business projects to remain relevant in the face of changing external circumstances.

Business projects typically run to much tighter deadlines than IT projects and may have far fuzzier objectives. As an example, imagine that a business department needs to rapidly identify all clients in a particular segment, in order to respond to a competitor’s new product launch, or risk churn of this key segment. At the same time the legal department needs to identify all U.S. clients in order to comply with pending legislation.

Ad hoc data analysis of this nature is difficult for IT to deliver in time – typically each requirement would be queued with one user losing priority and missing their deadline. This has a negative impact on the business, and if the business user had been involved from the start IT would be better equipped to deal with requirements.

Specialist data quality platforms enable business users to take ownership of data governance and data quality issues, either on an ad hoc basis or as formal projects that involve IT. Some of these specialist tools address the key criterion of enabling the business user to quickly deliver “good enough” data quality while supporting enterprise data volumes and standards and plugging in easily to enterprise systems such as ERP and MDM, either in real-time or via batch..

Specialists also bring the data centric view that is necessary for successful data governance and master data management. This experience, and the vendor’s ability to transfer this knowledge into the corporate environment, can add tremendously to an organisation’s ability to deliver relevant and sustainable data management capabilities – bridging the gap between business and IT.

Data quality is recognised as a key driver for the successful delivery of data governance, master data management and BI projects, and is a key component of many compliance and regulatory projects.

Data quality and BI are intrinsically linked, and the data quality space can be expected to follow the predictions around BI for the coming year. If 2012 is the year of delivery for BI tools, it stands to reason that data quality will follow suite – there is no reason why 2012 should not be a year of delivery all year round.

According to their research, leading firms are recognising the importance of data governance to the success of many other initiatives, including master data management. As a result, data governance is reemerging as a key focus for 2012.

The research also shows that data governance, data quality and master data management initiatives require a joint business and IT commitment for success. This is still a lesson that most companies need to learn – but leading companies are now recognising the critical nature of data governance for successful enterprise data and process initiatives to succeed.

In South Africa, Data Governance emerged primarily in the banking industry with a focus on Risk Management. These early attempts have struggled to gain buy in from the broader business – possibly because they were perceived to be too focussed on Risk – to the exclusion of other business areas. Also, these implementations have struggled to show measurable returns due to a lack of investment in Data Quality and Data Governance tools that can support the business community. Unfortunately, spreadsheets and SQL scripts do not enable business users to assess whether data governance is improving enterprise data management.

We have seen a shift in this position in our client base – in banking and other industries.

One of our clients, a global mining company, has deployed Trilium Software’s data profiling technology ( with a few days of skill transfer to the user) to automatically measure and manage data quality affecting their weekly HR reporting. During the course of 2012 they will deploy similar solutions in other areas in support of their data governance strategy.

In financial services we have seen a shift away from a risk centric view towards a data centric view. Our clients are recognising that data that is being used for more that one purpose must be measured against the needs of all key role players – and are using Trillium Discovery to achieve this.

It boils down to a practical application of data governance to address an immediate business need. “We know we have data quality issues that are affecting key business processes. We don’t know what these are so we don’t know where to begin to fix them!”

Simple metrics that support the business, delivered with a tool that can be used by the business, allow the second question to be answered. Business can immediately establish where to focus attention, do root cause analysis, and re-mediate idenitified issues. More importantly, in a tight economy business can prioritise on those areas where they will get the best return by making fact based decisions, rather than guessing where the problem areas are!

This post was originally published in Data Quality Matters blog

Another friend, after adding some photographs to her profiles was freaked out by the question – “Did you take these at Bob’s wedding in Nieu Bethesda?”. She had – how did they know?

These kinds of analyses are examples of the emerging trend towards big data – in particular embracing the data generated by social media – in order to drive competitive advantage. According to McKinsey Research the use of this kind of data in a retail space can increase the operating margin by as much as 60%.

Yet bringing this kind of data into an enterprise creates serious challenges. According to the IDC there is already more data than there is space to store it. Approximately 90% of the data that is being created is unstructured – data that does not arise from transactional activities or fit into a relational database. Some examples would include email, video, spatial data, documents and spreadsheets.

According to IBM, the volume of ALL data that was created before 2003 is now being recreated every two days. The challenge for most organisations will be to separate the wheat from the chaff – of the myriad objects created daily how will you filter out those that have value (or must be retained and protected for legal purposes) from those that are garbage.

Your data  governance program should be planning for this challenge already.

Do you have a plan for managing unstructured data?

Can you you answer questions such as: “Where is our critical data stored? Who created it? Who has access to it? Should they have access to it? Which of this data is redundant or stale and which of it must be kept?”

Your ability to answer this kind of question will be impacted by both your governance policies (what are yourpolicies for big data?) as well as your technical ability to handle large volumes of data using data governance tools .

Of course, knowing where it is is one thing. The other challenge that must be overcome is addressing the quality of these huge volumes. A simple example, my friend had been engaged for months prior to updating her status on facebook – by the time the wedding planners got her in their sights she was practically on her honeymoon. The volumes of data combined with the untrusted nature of the source mean that much of the data received is of no value.

Your business need to decide how relevant each potential source of data is to your competitive advantage. Yes, big data can add real value, but the business that is not buried under a mountain of garbage may be able to make better decisions than one that is. Data quality will become even more critical as a differentiator as big data becomes pervasive.

This post was originally published in the Data Quality Matters blog

It raises an interesting point. Ambiguity in data can have life or death implications.

In most companies data volumes prohibit manual exception management of ambiguous data – particularly for matching.

Where data volumes are lower, staff complements tend to be low too. So 70 or 80 exceptions a month may be too many for a small business to deal with.

Automated matching technologies are an extremely useful tool for identifying duplicate records in your client base, inventory,  supply chain, or elsewhere.

The question that arises is how to ensure that ambiguity is dealt with correctly!

There are two extremes.

At one end you do not match any ambiguous records. “J. Smith” may or may not be “John Smith” – so you continue to treat these as two seperate clients.

On the other other extreme you would match all ambiguous records – of course “J. Smith” must be “John Smith”!

False positive matches – where you incorrectly assume that two separate entities are the same – expose your business to far more risk than the other possibility – that you incorrectly continue to treat the same entity as more than one individual. Which is worse: that as a client I have two account numbers and receive two invoices, or that as a client I cease to exist in your dataset and am never invoiced?

Any matching process that does not first standardise data to remove ambiguity is prone to false positive matches that may never be picked up.

The key requirement for matching is to break data into its elements – e.g. a product description may be made up of a the BRAND, the UNIT OF MEASURE, the MATERIAL and the COLOUR. Each of these elements should be standardised to the extent necessary to remove ambiguity – is “H2O too” really the same as “H2O2″? (NO!).

Where ambiguity remains it is better to err on the side of caution – merging to client or product records incorrectly can have catastrophic consequences and be almost impossible to undo. With experience you can quickly identify which are the key elements of an object and focus your effort on standardising these only.

This gives you data certainty that the matches that have been successful can be ignored as exceptions – and allow you to focus you attention on dealing with the much smaller subset of records that have been failed as possible matches (if this is important). This will reduce your operational data management costs and will substantially increase you return on investments in MDM, CRM or similar “single view” technologies.

This post was originally published in the Data Quality Matters blog

As data management professionals, we all understand the importance of metadata. The problem we face is that the concept of metadata – information about information – is all encompassing and  means different things depending in the context and who you are talking to.

IT Architects argue eloquently for the need for  a consolidated metadata repository. However, a single repository that holds everything from the business definition of a terms (“How do we calculate GROSS PROFIT”), to a dictionary of data defintions (“Where and how do we store Product codes?”), to active data statistics (“How many Null values do we have for Client ID in the Orders table?”) is neither technically practical nor, in my opinion, particularly useful. The volume of work required to capture this information in a single place means that it will be out  of date practically as soon as it starts – and it will never be completed.

At the very least, the practical implementation of such a repository will require significant rework – a lot of metadata is stored already – as ER diagrams, in process models, as data flows or in your Data Profiling tool.

For me the problem boils down to the fact, as discussed in Jim’s post The Metadata Continuum, that metadata in itself has no standards.

I would like to suggest that we, the data management community,  need to be a bit more specific and a bit more pragmatic if we want to get business support. If we are trying to create consensus as to the “calculation of Gross Profit” or “the definition of a Customer” maybe these (and similar concepts) can be categorised as our Business Glossary (say)?  Maybe our Data Quality rules are classified as just that – Data Quality rules?

In my opinion, business may be interested in ensuring that all BI reports calculate Gross Profit using the same formula, and as result show consistent values. They probably don’t care about a metadata repository that will link every piece of information (at every level) in the business.

By breaking the metadata challenge down into smaller pieces we can make it more achievable and may find it easier to get buy in from business stakeholders.  At the very least – hopefully – we will all be talking about the same thing!

The problem, in most cases, is that the business benefits of data governance have not been clearly articulated.  In many cases, data management staff may not know where to start to identify potential benefits.

A few obvious candidates stand out:

1. Corporate Governance  

Following the Enron scandal the US government took action to hold directors accountable for the misrepresentation of financial results with the Sarbanes-Oxley Act. Many South African companies have dual listings or international partners and have to comply with this act.  On the local front, King III is the most comprehensive set of guidelines yet published to ensure good financial governance for South African companies, while government entities must comply with the Public Financial Management Act (PFMA).

Are the underlying data management practices in place at your company sufficient to guarantee the accuracy of your financial results? How much stock are you holding? How much bad debt? If you have a financial governance project running this would be a good place to look for a business case.

2. Privacy legislation

The Consumer Protection Act, the PCI-DSS (for credit card vendors) and the pending Protection of Personal Information bill all require that sensitive data is secure, is of good quality, is accessed on a need to know basis only, and is disposed of when no longer needed.

Can your business accurately identify all instances of data that you hold for each client? Which is the most up to date? Who has access to it and do they need to? Sensitive client data tends to spread across the enterprise into various departmental and product systems and managing this challenge is a governance problem.

3. Risk

Basel II (and the pending Basel III) and Solvency II are regulations set up to manage and assess risk for banks and insurance companies. Organisations are required to establish a process for data quality management and account for adjustments to historical data. The regulations require that risk calculations must be “provably correct” based on ongoing assessment and monitoring of core risk data.  Like any large program that touches all departments in the business a pragmatic data governance implementation will add value.

Compliance driven business cases are built around the avoidance of penalties and fines, the reduction of financial and reputational risk and the cost savings that governance can bring to tasks that will have to be performed anyway.

4. Master Data Management (or other data intensive IT projects)

Master data, by definition, is used for many purposes and across many areas of the business. Data governance helps to ensure that the MDM implementation takes all the necessary views and uses into account, helps to manage conflict and resolve issues.

Data governance (and data quality) are recognised as critical success factors for MDM projects – why spend hundreds of millions on new systems without planning for success?

5. Duplicated effort

In many large organisations we see huge duplication of effort as different teams try to address the same issue. At worst this may simply result in wasted expense – multiple projects trying to achieve the same result. In the worst case, projects may clash with each other – with one undoing the results of another.

One real example. A project team was set up to capture missing rate payer information for billing purposes. Another project was set up to remove rate payers without a valid postal address. The result – one person capturing data and another, two cubicles away, deleting the records as quickly as they were captured.

A major benefit of data governance is the business alignment function – ensuring that all projects that will impact data are understood and coordinated across the enterprise. This can drive significant cost savings.

Ultimately your business case may start with one of these, or with something else entirely. to find specific business cases you need to talk to these project teams and identify issues that can be related directly to data management.

I would recommend starting small – where can you deliver measurable benefits within your existing budget and capacity? Business buy in will come as you show value – remember to measure and communicate what you are achieving.

Please share your experiences – where has your governance program delivered value? Where did you find traction and how did you sell it?