Lobue News: Nov 01, 2012

LoBue Real Spotlight - 4Q 2012: Big Data

Big Data: Big Results or Big Confusion?

Big Data:  The latest ‘buzz-phrase’ gathering momentum within financial services organizations and the financial press. This one however is unlikely to disappear anytime soon. So brace yourself—you will be hearing a lot more about Big Data!

As the term embeds itself into the lexicon of bankers and insurers globally, the meaning is already being warped and shaped to suit the user. Simply put, Big Data is data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the strictures of existing database architectures. To gain value from this data, you must choose an alternative way to process it.

The reason the concept is so important in Financial Services, is that Bankers and Insurers have never been so overwhelmed with torrents of data from multiple sources as they are now. Sources include customer data, process data, business performance data, staff data, channel data, and data from external sources, such as web, social media, email, SMS, call data records, etc.  This data explosion can reach beyond terabytes into petabytes for an individual organization. The sources and volumes of data are increasing exponentially, as is the pressure on individual users to achieve deep insight into the data and make the smartest decisions possible. Add to this the business requirements of ultra-high speed loading of this data, as well as search, discovery, and storage capabilities to meet government and regulatory requirements: most organizations are just not prepared.

So what’s happening in practice? Management staff in all sectors around the globe are struggling to make sense of the increasing volumes of data, spending less time actually managing their businesses, and measuring performance inconsistently based upon evolving and new data types and sources.

Can managers realistically hope to distill or filter these data torrents intelligently, to ‘see around corners’ that their key business rivals cannot, or is this a battle that simply can never be won, fighting on current terms at least?  A brief look into how others are approaching the challenge may help in clarifying the issues.


Big Data: Where’s the important information?

A recent extensive research exercise* surveying over 500 IT professionals working in multiple sectors throws some light on the changing dynamics of information. If we separate information broadly into two types—structured (e.g. data within a core banking system, or CRM module), and unstructured (e.g. emails, web data, CDRs, SMS, audio and video files, ad-hoc documents etc.), whilst the volume of both types are growing rapidly, it is the availability and use of unstructured data that users are experiencing explosive growth in.  When asked how things are changing, respondents are equally split, half are expecting more business critical data to come from unstructured sources and the other half expecting structured sources.  This illustrates that Big Data will not be sweeping away the very strong position that relational databases and other established solutions hold in the near term. With so much data out there, just how well are organizations exploiting what they already store?


Big Data: How well is it currently being used?

The survey showed that roughly one in eight organizations fully exploit their structured data while an even smaller fraction does so for unstructured data. Taken together, these results show that there is obvious potential for both ‘traditional’ business intelligence/data analytics tools and more cutting edge Big Data solutions to be utilized for the benefit of very many organizations.

Indeed, the fact that very large numbers of respondents indicate they could do better in terms of exploiting the information they hold can be interpreted as both a recognition of existing business pressures to get more value from their data as well as an acknowledgement that, until now, BI has been seen in many organizations as the domain of a few specialized analysts rather than being something that could help steer the company in its daily operations.

But readers of REAL should not walk away with the idea that the days of structured information are numbered. The research actually proves otherwise.  The prevalence of ‘core data systems’ in banking and insurance particularly is extremely wide and ingrained.


Big Data: So how do we make sense of our environment?

Data volume and sources are exploding, particularly unstructured sources, but it seems traditional repositories are here to stay and often persist in holding an organization’s ‘crown jewels’ in terms of data quality.  At the same time, now is an opportunity to take advantage of additional data sources and capitalize on a new wave of management tools, ahead of competitors. 

LoBue understands the challenges and, with multiple clients, have deployed programs to distill inbound information to that best suited to assist management to focus on their ultimate business objectives.


*Freedom Dynamics—The Register (Sept. 2012)