- Part 1 - Introduction
- Part 2 - Spell out your ambitions
- Part 3 - Select the organizational structure
Step 3: Perform a proof-of-concept
As with most new endeavors, it is difficult to dive head-first into Data and Analytics without first implementing a scale experiment to ensure you have the correct people, technology, leadership support, and business interest. That said, a lot of companies dip their toes into analytics, find minimal value, and then slow down or back away from the initiative. In some cases, this could be a perfectly appropriate response. However, it is sometimes a reaction to a poorly conceived proof-of-concept (PoC). When planning the use case and scope for your Data and Analytics PoC, be sure to consider the following.
- Spell out your goals, document what success looks like, and develop a methodology to measure and report on progress towards that success state. Things can evolve along the way, but you need a foundation upon which to build so you are not just throwing things at the wall to see what sticks.
- Select an appropriately sized use case such that it can be completed in a reasonable amount of time. You want something significant enough to provide useful output but small enough that it is achievable in the time allotted. That time can vary, but 3 months is not unreasonable to implement something new and fairly complex with a high potential value.
- Take the opportunity to build support by selecting a use case of interest to business leadership instead of focusing solely on the technology and technical leadership. The more colleagues who have a stake in the success of the PoC, the more likely they are to contribute expertise, and the more likely the PoC is to be successful. Do not implement the PoC in secret and then spring the results on the business in an I just figured out how to do your job better approach.
- If return on investment (RoI) is an important consideration for your Data and Analytics program, select a use case with notable, calculable time saving or monetary saving/earning numbers. This is not to say that the PoC is going to directly save time or save/make money, but you should be able to clearly show how it could lead to that when put into production.
- Analyze your company's current state of data readiness and scope the use case appropriately. This can be a tough one. If your company already has a solid platform in place to handle day-to-day operational reporting and basic analytics needs, then the PoC should demonstrate the potential to do more with advanced analytics like machine learning. If your company is still struggling with the basics, a PoC that overshoots the current state by too much may not be the right choice. Such a PoC would tend to have one of two impacts. First, it could result in the question of why would the company invest in something so far-reaching before it solves for daily needs. Second, it could get colleagues and leadership excited about a future they think is just around the corner when it is actually years away and get you into an over-promise and under-deliver situation. That said, you do want to engender a degree of excitement to garner support. So, finding a balance is key.
- If you do opt for a PoC focused on forecasting, machine learning, or other advanced analytics, be sure to gather enough data to actually find actionable insights. There may be hundreds or even thousands of variables to consider before you can pare things down to a reasonable size and still provides business value. Do not go into the PoC thinking that you are going to build a machine learning model on a couple dozen variables. You are likely to be disappointed. Also, be prepared for the possibility that your first endeavor may not provide significant insights or it may just confirm commonly held beliefs. What you learn through the process is still important, and you can also pivot to something more promising. Communication and transparency is key.
- Do not miss the forest for the trees by seeing the platform as only capable of serving a single or limited set of purposes. Step back and consider the possibilities. A correctly architected Data and Analytics platform can do all manner of things. I have listed a few below, and it is important to figure out what will best speak to the leadership of your company.
- Analytical insights – Use the data to find patterns, forecast the future, and prescribe actions with the highest potential success rates.
- Process offloading – Use the massive compute power and inexpensive storage inherent to the platform to offload long-running jobs currently performed by other platforms.
- Active data archiving – Store all data, forever, on a platform where storage is cheap and the data is easily and securely available for analytics and reporting
- Real-time alerting - Process data in near real-time and produce proactive alerts when a non-conformance is found or a specific state is reached.
- Self-service – The biggest current trend in Data and Analytics is the democratization of capabilities so that users are empowered to help themselves instead of waiting in line for their needs to be prioritized and addressed
- If at all possible, use platform (PaaS) and software (SaaS) service offerings available from cloud providers like Amazon, Google, IBM, Microsoft, etc. This is particularly true in a PoC environment that you want to build out rapidly and then dispose of or possibly use for future experimentation. However, the cloud also provides significant benefits when moving to the production phase. People greatly underestimate just how difficult it is to build, maintain, and tune something like a Hadoop cluster either on premises or in cloud infrastructure (IaaS). It is extremely time-consuming and can be very expensive. Cloud services allow you to focus on satisfying business needs instead of fighting with the technology while being easy to expand as demands increase. Done correctly, the cloud can also be significantly less expensive. If your company does not currently allow it, find some partners in the organization, build a business case, and fight for it as the benefits are huge.