If your company is in the enviable position of growing rapidly and needing to scale their IT organization, there are a number of ways to expedite the process while ensuring that you get the right people on board and integrated into the team as quickly as possible.
First, we will take a look at a couple examples of what not to do. These stories are based upon first-hand experience
Avoid Accuracy through volume
It might be a good practice if you are a machine gunner, but accuracy through volume in hiring is a difficult and often disruptive undertaking. I worked for a company years ago that would hire multiple dozens of new college graduates each year. On one hand, it led to a lot of variety of thought and fresh, eager colleagues who could sometimes be a breath of fresh air at an otherwise old school and slow-moving company. On the other hand, the very high turnover and constant on-boarding where disruptive to the larger team and led work to be completed more slowly and less efficiently than it could have been with a more streamlined and well-planned team. New college hires can be a tremendous addition to the team, but they need to be balanced by experienced colleagues who can help achieve the first "what to do" item below to give the best chance of a new hire quickly becoming a productive member of the team who then stays with the company and becomes one of the experienced colleagues to keep the cycle going.
Avoid unattainable job requirements
It is very important to carefully consider what you are looking for in a candidate whether you are hiring just one or many. However, this can lead you to put together a job description with requirements so specific that no candidate will fill the bill. I sometimes like to refer to this as IT folks making it more difficult for themselves and others in IT to find a job because the IT folks are the ones building the algorithms that process resumes. When the number of requirements put into those algorithms is too high, a lot of resumes for excellent candidates can be rejected simply because of a missing keyword. There is certainly a base level of technical skill necessary to fulfill any IT role. However, I prefer to look for candidates with certain types of experiences like taking something from inception to production and an ability to passionately tell the story of interesting things they have worked on. Even a college recruit should have something they are passionate about though it might not be something technical. In all my interviews, I always ask the candidate to tell me, with a significant level of detail, about something interesting they worked on. During their answer, I pay attention to both their passion and their level of detailed knowledge on the subject. Even if the work is different from that of the role for which they are applying, a candidate who was passionate and picked up and utilized detailed knowledge in the past can do the same again for your company.
With those two examples of what not to do, we will turn our attention to a number of "what to do" items that will help find the right candidates and get them integrated into the team as quickly as possible.
Nail down your corporate culture AND your IT culture
Most companies of any significant size have a culture that is frequently carefully cultivated by HR and corporate leadership. This is definitely a great start, but there is more to be done in this area. Every company also has a culture, hopefully aligned with the overall culture, within IT. This is a culture of both soft and technical skills that needs to be cultivated by IT leadership. A few examples of key items to figure out include technology stacks, release schedules, 24/7 support, team composition, cross-team collaboration (development, operations, infrastructure, QA, etc.), knowledge sharing, colleague recognition, celebrations / postmortems, and the role of IT around the ultimate deliverables of the company. Beyond that, IT colleagues need to understand how their personal work is contributing to the success of both IT and the company. The more rapidly the IT department needs to grow, the more closely the IT culture needs to be monitored, refined, and even documented. When a candidate is partaking in an interview, the colleagues performing the interview should be able to provide a very detailed and uniform view of the culture so that all parties can determine if the candidate is a good fit. When a new hire starts, they need to be quickly ingrained into the culture to become a contributing member of the team as soon as possible. I have spoken with many people about and experienced myself instances of still not feeling like a part of the team and the IT culture even after 6 months of employment. The overall company culture may be clear, but a colleague may still have difficulty with all the nuances within IT. In the best case, such a colleague is a less productive member of the team. It the worst case, they have a ton to offer but leave to find a new opportunity where they can feel more supported as an important part of the team.
Generally speaking, many IT teams rely too heavily on HR and corporate leadership to drive the culture and forget that it is equally as important to define and refine the culture within IT and the key items that make IT unique.
Decide the right team composition in advance
Team composition to ensure the right assortment of senior and junior experience, the ratio of management to individual contributors, and diversity of thought is very important. Any guidelines can be revisited, and you certainly do not want to eliminate a great candidate who may come along at any time, but these topics need to be carefully considered and documented as you begin to scale the team.
Get the company name out there
If a company wants to attract top talent, they need to get their name out and associated with positive connotations. This can be tricky when it comes to IT if the company's primary product is not IT services. So, the company overall should ensure that it makes a positive name for itself, but leadership and key contributors from IT need to also help on the technical front. A highly technical candidate will generally prefer a company they see as a) meeting and exceeding their level of technical acumen and b) valuing technology as a significant contributor to corporate success. The key is to give a face (more likely multiple faces) to the company at meetups and other more formal technical events. Attending and participating are fine, but sponsoring, presenting at, and even hosting will go a lot further towards cultivating the company's external, technical image. Certainly, a company needs to exercise caution to ensure that the right message is getting across to consistently highlight the best attributes. This goes back to an earlier bullet about cultural alignment above. If a company does not put themselves out there to build a reputation as an appealing place for a technologist to work, though, it will be very difficult to attract the best talent available.
Partner closely with recruiting
Assuming that your company has a recruiting department, partnering very closely with these colleagues is very important, They need to be brought into the fold of not just the corporate culture but the IT culture and the other bullet points noted in this article. I have seen many instances where IT simply creates a job description, passes it along to recruiting, and then complains that they are not getting the right candidates. Typically, a recruiter is responsible for many areas of the business or, if the company is large enough, even many areas just within IT. Working with the recruiter as a trusted partner and aligning very closely is necessary to find the right talent quickly. Beyond the job description, you need to define the personality, experiences, and other less-tangible traits for which you are looking. As candidates pass through, you need to be very thoughtful and thorough with the feedback provided to the recruiter whether the candidate was hired, offered and declined, or passed. There are many things to learn from each scenario, and it is important to share that knowledge both ways between IT and recruiting.
Work with organizations who place military veterans
This item is fairly obvious but often overlooked. Studies show that military veterans, at a higher than average rate, make for very productive and dedicated students, employees, and leaders based upon their experiences in the military. I have personally worked with a good number of veterans over the years and would not hesitate to hire more in the future.
Recruit from within
Recruiting from within comes in two flavors. One is fairly obvious and involves looking at existing employees to fill open roles. This can mean a promotion, a lateral move from one technology stack to another, or even a move from outside of IT. One of the best colleagues who ever worked for me came from outside of IT and rapidly worked his way to Director of Enterprise Architecture within fairly large IT organization. I hired him away from our original company and would be more than happy to hire him again in the future. Keep an open mind when looking within. It is a great way to keep colleagues happy through variety and challenge and to help a high performer spread their positive influence to other areas of the company.
The second flavor of recruiting from within is to mine the networks of your existing colleagues. If you are doing a good job of getting you colleagues ingrained into the culture and being happy and productive, then they should already be reaching out to friends and former co-workers to bring them into the fold. Extra incentives can but put into place to aid in this process, but I tend to disagree with significant, monetary rewards except for difficult to fill, critical roles. You do not want colleagues just using the opportunity to make a little extra money. Instead, recognition and non-monetary rewards can be used to influence participation.
Consider contract-to-hire
Sometimes companies forget about this option. Although it can be inconvenient to have new colleagues who may leave after a 6-month contract, it provides a good option for both sides to "try before they buy". If your company does not quite have the other items mentioned in this article in place, you may find really solid candidates willing to give it a try via a contract without signing up for a long term commitment. The key with these colleagues is that you should treat them just like any other member of the team because your goal is to bring them on full-time. If you isolate them from the full-time colleagues or otherwise treat them differently, the likelihood they will make the transition is greatly reduced.
Tuesday, November 12, 2019
Wednesday, November 6, 2019
7 Positive Traits of A Great Leader
In this post, I will share my thoughts on 7 positive personality and professional traits that make a great leader in no particular order. The reason I have not specified an order or numbered them is because every leader should strive for all 7. Convincing the reader to adopt some subset of the 7 is not the purpose of this article. Along the way, I will also throw in a few personal stories to highlight my points. Please note that these stories are not intended to be a statement about how "awesome" my leadership abilities are. I am simply pointing out instances where I have tried my best to embody these traits and be the best leader that I could in different circumstances. Perhaps I will write another post covering the mistakes I have also made along the way because I, being human, am certainly not perfect.
A great leader...
Practices radical transparency and honesty
You can read up on the concept of Radical Transparency on any number of websites. For the purposes of this article, I am primarily referring to the fact that a leader's job is to shield his or her team from chaos, politics, priority changes, budget snafus, etc. It is not to shield them from information. Everybody involved in a work relationship is an adult and deserves to be treated as such. No matter the information, sharing and discussing it in a controlled way is far superior to the alternative because the team is always going to find out regardless of any attempts to prevent it. Certainly there are times when a leader is simply unable to share information, and this is where the honesty also kicks in. When the team inevitably asks the leader what they know of a given situation, it might be easier to respond that they know nothing. Honesty, though, calls for the leader to let it be known that they have knowledge they are currently unable to share. He or she should explain why and then work to get to the point when they can share more as soon as possible.
A leader should both practice transparency and honesty and expect them from his or her colleagues. They must be willing to accept honest thoughts and opinions even if they differ from their own or provide constructive criticism. It is important to build a culture where such interactions are openly accepted and encouraged while also striving to be tactful in the approach (i.e. do not just call out somebody in the middle of a meeting).
I have experienced a few occasions in the past where my level of transparency and honesty have been questioned for being possibly being too "extreme". This typically comes from old-school leaders who do not believe in the same philosophy, but I can honestly say that, over 20+ years of working, I have never had an individual whose attitude, quality of work, happiness, etc. has ever been negatively impacted because I was transparent and honest with them. I have also never had another leader tell me that they wish I would provide less detail of the work my team is doing, the status of our platform, and the successes and roadblocks we have experienced because they do not want to know.
Leads from the front
A leader should never ask his or her team to do something they would be unwilling to do themselves. I am not talking about specific tasks as the leader is often not capable of executing tasks in which a member of their team specializes. I am talking about less tangible things. If a leader expects their team to work overtime, the leader should also be willing to put in the extra effort. If a leader expects their team to give up vacation time, the leader should also be willing to make the same sacrifice. If the leader expects the team to "sell" an initiative to other leaders, he or she should be right next to their team selling it and putting their own neck on the line if the initiative falters. A leader must be out ahead of his or her team doing their best push aside all the roadblocks, taking the same risks, and putting in the same effort. They cannot sit back and critique and second guess from behind.
When I worked for a large consulting company several years ago, I was in a technical and people leadership role that could have allowed me to sit back and tell people what to do while working a 9-to-5 schedule. Instead, I decided to take a very pro-active approach to the technical side by working ahead to figure out things that the team would need the next week or the next month so that I could provide guidance when we got to that point in the effort. I also helped the team execute upon all of the promises each agile sprint to ensure that everything was completed when it was an option to simply sit back and point out how specific team members had not completed their assigned tasks. This allowed me to get to know each team member, develop very solid relationships built on respect and trust, keep the clients happy, and, in the end, help to make everybody more efficient while improving work/life balance through better sprint planning. Leading from the front is definitely not the easy path, but it is the right path to take.
Is emotionally consistent
Every leader has a style, and colleagues can become accustomed to just about anything if that style is applied consistently every day. So, a leader must pick their style and stick to it versus setting up a Dr. Jekyll and Mr. Hyde situation where colleagues never know which leader is going to show up to a given meeting or interaction (in-person, phone, email). This is not to say that a leader can never make a change, and this goes back to transparency and honesty. If a leader sees an opportunity for self-improvement, he or she should be open with their team and bring them along on that journey.
More years back than I care to acknowledge, I played high school football. I still vividly remember two of the coaches in particular. All of the kids were essentially terrified of one coach because he could shift from being friendly and supportive to being a raving, cursing lunatic in a matter of moments. We never knew who was going to show up for each practice. On the other hand, there was a coach that everybody liked who was 3/4 of the way to raving lunacy at all times. We all just acclimated to his style and knew exactly what to expect and how to keep him happy. I am certainly not advocating raving lunacy as a leadership style, but the key is that people can deal with consistent behavior even if they sometimes disagree with it. At the very least, it gives the leader the ability to consciously analyze themselves and how their style impacts their team in order to execute changes for the better over time in collaboration with their team.
Exercises reasonable empathy
This is the one point (the "reasonable" qualifier) that I believe may evoke some negative feedback in this article because empathy in leadership is a very hot topic right now. Empathy is, indeed, critically important both for work and personal life, but a leader needs to draw the line so that colleagues are not taking advantage by seeing just how much they can get away with. This is not for the benefit of the leader but for the benefit of the rest of the team who are not trying to take advantage. If they see a colleague getting away with things that they should not, it negatively impacts the rest of the team. This, however, is a very delicate situation and must be approached thoughtfully and cautiously to ensure that the leader is finding the right balance.
At a previous leadership role, I was given the unenviable task of writing a review for an under-performing colleague even though I had minimal experience with that colleague at the time. I was flat out told to write a negative review so that the colleague could be put on a plan and set up for subsequent termination. Exercising a reasonable amount of empathy called for me to interview other team members with whom the colleague had worked and particularly the colleague himself to understand the situation before blindly following instructions. During this process, I learned that the colleague had been asked to execute work that he was physically incapable of executing - not out of laziness or lack of caring but a literal inability to see the computer screen the same way the average person does. He was afraid, though, to admit this and risk negative repercussions, and the rest of the team was not really aware of the situation. After learning of this, I wrote up a much more reasonable review pointing out the issue and then spent the next few months mentoring him on the back-end development that did not require the same visual abilities as front-end development. This also meant re-organizing the team to account for the change in workload, and the team was willing to exercise reasonable empathy when they understood the situation. Through the hard work this colleague was willing to put in, the understanding and trust of the team, and some assistance from me, the colleague became a much more successful developer and contributor to the team.
Decisively builds consensus
Sometimes leaders describe themselves as "consensus builders" because they see themselves as open-minded, but they often forget an important piece to the puzzle. A leader certainly needs to be open to ideas and collaborative with colleagues of all levels, but they also need to then assimilate all of that information and form a decisive action plan to move forward. This latter part is often overlooked. Just being open and collaborative does not move the needle. A leader must get all parties on the same page and moving forward together. He or she cannot expect that the colleagues with varying opinions will somehow find a cohesive direction among themselves without the leader's further involvement. A leader must dictate the direction of execution without being a dictator. The action plan may not satisfy all parties, but integrating the top ideas into a cohesive plan based upon the current situation and knowledge is the best any leader can do.
Puts the team's needs first
For a leader, putting their team's needs in front of their own can take many shapes. Perhaps it is the leader putting himself or herself into an uncomfortable situation that goes against the grain because they know it is the right thing for the team. It can also be the leader openly celebrating the accomplishments of his or her team, inviting team members to leadership meetings to present those accomplishments, and generally acknowledging the team instead of taking the successes as their own. The same can be said for failures where the leader should acknowledge their own role while helping to rectify the situation versus throwing a team member under the bus. This is a huge topic with a lot of applicable scenarios, but the key is to adopt the "we" mentality and even the word itself when discussing the team's efforts. A leader cannot always do what is best for the team, but the team should always be a top consideration in every decision.
When a start-up for which I was the VP of Engineering was acquired, an issue arose when the acquiring company sent very low offers to all of the existing employees - myself included. Because the executives of the start-up had left as part of the acquisition, that meant I was the most senior leader remaining. After discussing with the team, we came up with a plan that each person would share with me what they desired to make. I, in turn, would craft a write-up for each colleague with my honest assessment of their skills, what they contributed to the team, and what their salary should be. My portion was done privately with the trust of the team that I would be fair and honest. I then flew to the headquarters of the acquiring company and negotiated the salaries and bonuses of all my colleagues. While they did not all get exactly what they desired, each received a salary that was at least 35% higher than the initial offers with appropriate, annual bonus potentials as well. Only at that point did I negotiate my own salary. From a humanity and leadership perspective, this just seemed like the right thing to do. I worked closely with and liked all of these colleagues and wanted to see them treated fairly so that we could continue our collaboration. From a personal perspective, it also did not make sense for me to negotiate my position while the rest of the team got frustrated and possibly left for other opportunities. A leader without a team is not really a leader.
Accepts that failure is always an option
Business and technology are changing rapidly and dramatically. It is not possible to create a perfect plan and then execute flawlessly on that plan. Accepting that failures will happen and iterating upon the best ideas formed using the most up-to-date knowledge at the time are key. A leader must be willing to both accept and admit mistakes within and outside of his or her team. The key when something goes wrong is to understand where things stand and to take decisive action to improve the situation while being honest and transparent along the way. Certainly, there are more considerations if a colleague is being intentionally malicious. Typically, though, what we might call a "mistake" in business is actually a decision that was made with the best intentions that seemed right at the time but turned out, in hindsight, to be incorrect. Everyone should learn from mistakes but what is the benefit in lingering on something that, say six months ago, seemed like the best decision? Even worse, what is the benefit of trying to assign blame six months later? Simply use whatever new knowledge has come to light since that time, build a consensus for the best approach to move forward, and take action under a new plan.
A great leader...
Practices radical transparency and honesty
You can read up on the concept of Radical Transparency on any number of websites. For the purposes of this article, I am primarily referring to the fact that a leader's job is to shield his or her team from chaos, politics, priority changes, budget snafus, etc. It is not to shield them from information. Everybody involved in a work relationship is an adult and deserves to be treated as such. No matter the information, sharing and discussing it in a controlled way is far superior to the alternative because the team is always going to find out regardless of any attempts to prevent it. Certainly there are times when a leader is simply unable to share information, and this is where the honesty also kicks in. When the team inevitably asks the leader what they know of a given situation, it might be easier to respond that they know nothing. Honesty, though, calls for the leader to let it be known that they have knowledge they are currently unable to share. He or she should explain why and then work to get to the point when they can share more as soon as possible.
A leader should both practice transparency and honesty and expect them from his or her colleagues. They must be willing to accept honest thoughts and opinions even if they differ from their own or provide constructive criticism. It is important to build a culture where such interactions are openly accepted and encouraged while also striving to be tactful in the approach (i.e. do not just call out somebody in the middle of a meeting).
I have experienced a few occasions in the past where my level of transparency and honesty have been questioned for being possibly being too "extreme". This typically comes from old-school leaders who do not believe in the same philosophy, but I can honestly say that, over 20+ years of working, I have never had an individual whose attitude, quality of work, happiness, etc. has ever been negatively impacted because I was transparent and honest with them. I have also never had another leader tell me that they wish I would provide less detail of the work my team is doing, the status of our platform, and the successes and roadblocks we have experienced because they do not want to know.
Leads from the front
A leader should never ask his or her team to do something they would be unwilling to do themselves. I am not talking about specific tasks as the leader is often not capable of executing tasks in which a member of their team specializes. I am talking about less tangible things. If a leader expects their team to work overtime, the leader should also be willing to put in the extra effort. If a leader expects their team to give up vacation time, the leader should also be willing to make the same sacrifice. If the leader expects the team to "sell" an initiative to other leaders, he or she should be right next to their team selling it and putting their own neck on the line if the initiative falters. A leader must be out ahead of his or her team doing their best push aside all the roadblocks, taking the same risks, and putting in the same effort. They cannot sit back and critique and second guess from behind.
When I worked for a large consulting company several years ago, I was in a technical and people leadership role that could have allowed me to sit back and tell people what to do while working a 9-to-5 schedule. Instead, I decided to take a very pro-active approach to the technical side by working ahead to figure out things that the team would need the next week or the next month so that I could provide guidance when we got to that point in the effort. I also helped the team execute upon all of the promises each agile sprint to ensure that everything was completed when it was an option to simply sit back and point out how specific team members had not completed their assigned tasks. This allowed me to get to know each team member, develop very solid relationships built on respect and trust, keep the clients happy, and, in the end, help to make everybody more efficient while improving work/life balance through better sprint planning. Leading from the front is definitely not the easy path, but it is the right path to take.
Is emotionally consistent
Every leader has a style, and colleagues can become accustomed to just about anything if that style is applied consistently every day. So, a leader must pick their style and stick to it versus setting up a Dr. Jekyll and Mr. Hyde situation where colleagues never know which leader is going to show up to a given meeting or interaction (in-person, phone, email). This is not to say that a leader can never make a change, and this goes back to transparency and honesty. If a leader sees an opportunity for self-improvement, he or she should be open with their team and bring them along on that journey.
More years back than I care to acknowledge, I played high school football. I still vividly remember two of the coaches in particular. All of the kids were essentially terrified of one coach because he could shift from being friendly and supportive to being a raving, cursing lunatic in a matter of moments. We never knew who was going to show up for each practice. On the other hand, there was a coach that everybody liked who was 3/4 of the way to raving lunacy at all times. We all just acclimated to his style and knew exactly what to expect and how to keep him happy. I am certainly not advocating raving lunacy as a leadership style, but the key is that people can deal with consistent behavior even if they sometimes disagree with it. At the very least, it gives the leader the ability to consciously analyze themselves and how their style impacts their team in order to execute changes for the better over time in collaboration with their team.
Exercises reasonable empathy
This is the one point (the "reasonable" qualifier) that I believe may evoke some negative feedback in this article because empathy in leadership is a very hot topic right now. Empathy is, indeed, critically important both for work and personal life, but a leader needs to draw the line so that colleagues are not taking advantage by seeing just how much they can get away with. This is not for the benefit of the leader but for the benefit of the rest of the team who are not trying to take advantage. If they see a colleague getting away with things that they should not, it negatively impacts the rest of the team. This, however, is a very delicate situation and must be approached thoughtfully and cautiously to ensure that the leader is finding the right balance.
At a previous leadership role, I was given the unenviable task of writing a review for an under-performing colleague even though I had minimal experience with that colleague at the time. I was flat out told to write a negative review so that the colleague could be put on a plan and set up for subsequent termination. Exercising a reasonable amount of empathy called for me to interview other team members with whom the colleague had worked and particularly the colleague himself to understand the situation before blindly following instructions. During this process, I learned that the colleague had been asked to execute work that he was physically incapable of executing - not out of laziness or lack of caring but a literal inability to see the computer screen the same way the average person does. He was afraid, though, to admit this and risk negative repercussions, and the rest of the team was not really aware of the situation. After learning of this, I wrote up a much more reasonable review pointing out the issue and then spent the next few months mentoring him on the back-end development that did not require the same visual abilities as front-end development. This also meant re-organizing the team to account for the change in workload, and the team was willing to exercise reasonable empathy when they understood the situation. Through the hard work this colleague was willing to put in, the understanding and trust of the team, and some assistance from me, the colleague became a much more successful developer and contributor to the team.
Decisively builds consensus
Sometimes leaders describe themselves as "consensus builders" because they see themselves as open-minded, but they often forget an important piece to the puzzle. A leader certainly needs to be open to ideas and collaborative with colleagues of all levels, but they also need to then assimilate all of that information and form a decisive action plan to move forward. This latter part is often overlooked. Just being open and collaborative does not move the needle. A leader must get all parties on the same page and moving forward together. He or she cannot expect that the colleagues with varying opinions will somehow find a cohesive direction among themselves without the leader's further involvement. A leader must dictate the direction of execution without being a dictator. The action plan may not satisfy all parties, but integrating the top ideas into a cohesive plan based upon the current situation and knowledge is the best any leader can do.
Puts the team's needs first
For a leader, putting their team's needs in front of their own can take many shapes. Perhaps it is the leader putting himself or herself into an uncomfortable situation that goes against the grain because they know it is the right thing for the team. It can also be the leader openly celebrating the accomplishments of his or her team, inviting team members to leadership meetings to present those accomplishments, and generally acknowledging the team instead of taking the successes as their own. The same can be said for failures where the leader should acknowledge their own role while helping to rectify the situation versus throwing a team member under the bus. This is a huge topic with a lot of applicable scenarios, but the key is to adopt the "we" mentality and even the word itself when discussing the team's efforts. A leader cannot always do what is best for the team, but the team should always be a top consideration in every decision.
When a start-up for which I was the VP of Engineering was acquired, an issue arose when the acquiring company sent very low offers to all of the existing employees - myself included. Because the executives of the start-up had left as part of the acquisition, that meant I was the most senior leader remaining. After discussing with the team, we came up with a plan that each person would share with me what they desired to make. I, in turn, would craft a write-up for each colleague with my honest assessment of their skills, what they contributed to the team, and what their salary should be. My portion was done privately with the trust of the team that I would be fair and honest. I then flew to the headquarters of the acquiring company and negotiated the salaries and bonuses of all my colleagues. While they did not all get exactly what they desired, each received a salary that was at least 35% higher than the initial offers with appropriate, annual bonus potentials as well. Only at that point did I negotiate my own salary. From a humanity and leadership perspective, this just seemed like the right thing to do. I worked closely with and liked all of these colleagues and wanted to see them treated fairly so that we could continue our collaboration. From a personal perspective, it also did not make sense for me to negotiate my position while the rest of the team got frustrated and possibly left for other opportunities. A leader without a team is not really a leader.
Accepts that failure is always an option
Business and technology are changing rapidly and dramatically. It is not possible to create a perfect plan and then execute flawlessly on that plan. Accepting that failures will happen and iterating upon the best ideas formed using the most up-to-date knowledge at the time are key. A leader must be willing to both accept and admit mistakes within and outside of his or her team. The key when something goes wrong is to understand where things stand and to take decisive action to improve the situation while being honest and transparent along the way. Certainly, there are more considerations if a colleague is being intentionally malicious. Typically, though, what we might call a "mistake" in business is actually a decision that was made with the best intentions that seemed right at the time but turned out, in hindsight, to be incorrect. Everyone should learn from mistakes but what is the benefit in lingering on something that, say six months ago, seemed like the best decision? Even worse, what is the benefit of trying to assign blame six months later? Simply use whatever new knowledge has come to light since that time, build a consensus for the best approach to move forward, and take action under a new plan.
Friday, October 25, 2019
Data Governance vs. Information Governance
A lot of folks have heard the terms "data governance" and "information governance", and the two terms are often used interchangeably. This, though, is a mistake as there are significant differences between the two initiatives. In this post, we will explore those differences and explain why it is important for companies to implement both.
To get started, it makes sense to spell out a few definitions. In the world of data, there is a progression from data to information to knowledge, and it is important to understand each of these.
Data is essentially just the raw data available across various sources like databases, spreadsheets, etc. For transactional purposes, it is obviously necessary as it underlies every platform out there whether it be an ERP, CRM, WMS, etc. Without the system on top of it, though, the data is not useful as you typically cannot learn anything from the data in its raw state. Data can flow among multiple systems and can be aggregated into a single location like a data lake.
Information is the cleansed and organized form of the aforementioned data. This is a very important step in the process that is often minimized by people looking in from the outside of a Data and Analytics implementation. You cannot go directly from data to knowledge without a significant amount of work spent turning the data into information. You can expect that 50% or more of the effort for the overall program will go into producing and organizing this information.
Knowledge is the set of insights gained from analyzing the information whether that be descriptive analytics where you are simply looking at what happened in the past to help you plan for the future or a fully executed neural network that is churning through huge volumes of data to prescribe actions to take. The key here is that these insights need to be actionable. Simply knowing something is not particularly useful if the knowledge cannot be acted upon to optimize or automate a process, create a new revenue channel, change a business model, etc.
This leads to my first argument for having separate data and information governance initiatives.
1) Data and information are very different from each other, and this difference is important to highlight and fully understand as it requires significant effort to move from data to information. Treating them the same tends to lessen the importance of the information in the eyes of the key stakeholders across the company while giving the very false impression that simply collecting data will lead quickly to knowledge.
Now that we understand the differences between data and information, we can take a look at who is responsible for each. Every company is different, but, generally, data governance is primarily the responsibility of IT. IT manages the platforms and databases, creates the integrations pipelines to move data around, pulls data into analytics platforms, etc. We will hold off discussing the details of implementing a data governance initiative and save it for a future post. At a high level, though, data governance is about privacy, security, risk management, compliance, and auditing. The work involved in such an effort mainly falls into the realm of technical solutions put together by IT. Of course, there is collaboration with the business (e.g. Finance) particularly around compliance and audit, but even in those cases technical solutions are typically required.
Note: Certain businesses like health care, banking, pharmaceuticals, etc. that tend to be very heavily driven by risk and compliance are likely to have entire teams dedicated to these areas. This might mean that a team outside of IT is actually driving those aspects of the project, but it is still appropriate to consider them part of the data governance initiative because the focus is on the data itself and not the information and knowledge that can be derived from the data.
Just about every company out there knows about the aforementioned aspects of data governance. They don't want people within the company accessing data they should not. They don't want data leaving the company and falling into the wrong hands. They must fulfill specific governmental and regulatory requirements in order to stay in business. Though not every company dedicates the time, effort, and funding required, the need is generally quite obvious.
Less obvious, though, is the need for information governance. Companies assign value to data by default when they decide that it is important to secure it. That is part of risk management - how much is it worth in dollars, reputation, etc. to have specific data stolen. However, there is also a huge amount of potential value in information that tends to go un-quantified. A lot of companies understand that they should use the data they produce to do something, but they have a difficult time building the business case and the large-scale implementation plan required to achieve what they need. This is where information governance helps. Though the initiative still requires IT support via the appropriate tools, information governance is much more business-centric. It focuses on data classification and metadata, data architecture and availability, data quality both via business and technical processes, data awareness and literacy, information life cycle management, stewardship, and value creation. As noted previously, there is some overlap with data governance, and there are certainly technical aspects typically handled by IT. However, the initiative is really about business colleagues helping to produce clean data by using their platforms correctly and then having access to that data with a high level of literacy and a sense of ownership so that they can ensure that it is of high quality and use it to create information, knowledge, and eventually value. Information governance is the framework that allows a company to assess their current state, create a plan for the future, and track progress towards the final goals. It is also a great basis for creating a business case for undertaking something like building a Data and Analytics program.
This leads to my second and third arguments for having separate data and information governance initiatives.
2) Information governance is often overlooked. Most companies have some sort of structured plan for addressing security, risk management, etc. around their data, but they frequently do not have a structured plan to address the actual use of that data to turn it into information and knowledge so that they can derive value. Splitting the initiatives ensures that ample focus can be applied to both.
3) There is a separation of duties required to focus on the appropriate aspects of governing data and governing information that can best be achieved by having IT run the data governance while folks on the business side run the information governance. Of course, there is some overlap and significant collaboration required to be successful.
To get started, it makes sense to spell out a few definitions. In the world of data, there is a progression from data to information to knowledge, and it is important to understand each of these.
Data is essentially just the raw data available across various sources like databases, spreadsheets, etc. For transactional purposes, it is obviously necessary as it underlies every platform out there whether it be an ERP, CRM, WMS, etc. Without the system on top of it, though, the data is not useful as you typically cannot learn anything from the data in its raw state. Data can flow among multiple systems and can be aggregated into a single location like a data lake.
Information is the cleansed and organized form of the aforementioned data. This is a very important step in the process that is often minimized by people looking in from the outside of a Data and Analytics implementation. You cannot go directly from data to knowledge without a significant amount of work spent turning the data into information. You can expect that 50% or more of the effort for the overall program will go into producing and organizing this information.
Knowledge is the set of insights gained from analyzing the information whether that be descriptive analytics where you are simply looking at what happened in the past to help you plan for the future or a fully executed neural network that is churning through huge volumes of data to prescribe actions to take. The key here is that these insights need to be actionable. Simply knowing something is not particularly useful if the knowledge cannot be acted upon to optimize or automate a process, create a new revenue channel, change a business model, etc.
This leads to my first argument for having separate data and information governance initiatives.
1) Data and information are very different from each other, and this difference is important to highlight and fully understand as it requires significant effort to move from data to information. Treating them the same tends to lessen the importance of the information in the eyes of the key stakeholders across the company while giving the very false impression that simply collecting data will lead quickly to knowledge.
Now that we understand the differences between data and information, we can take a look at who is responsible for each. Every company is different, but, generally, data governance is primarily the responsibility of IT. IT manages the platforms and databases, creates the integrations pipelines to move data around, pulls data into analytics platforms, etc. We will hold off discussing the details of implementing a data governance initiative and save it for a future post. At a high level, though, data governance is about privacy, security, risk management, compliance, and auditing. The work involved in such an effort mainly falls into the realm of technical solutions put together by IT. Of course, there is collaboration with the business (e.g. Finance) particularly around compliance and audit, but even in those cases technical solutions are typically required.
Note: Certain businesses like health care, banking, pharmaceuticals, etc. that tend to be very heavily driven by risk and compliance are likely to have entire teams dedicated to these areas. This might mean that a team outside of IT is actually driving those aspects of the project, but it is still appropriate to consider them part of the data governance initiative because the focus is on the data itself and not the information and knowledge that can be derived from the data.
Just about every company out there knows about the aforementioned aspects of data governance. They don't want people within the company accessing data they should not. They don't want data leaving the company and falling into the wrong hands. They must fulfill specific governmental and regulatory requirements in order to stay in business. Though not every company dedicates the time, effort, and funding required, the need is generally quite obvious.
Less obvious, though, is the need for information governance. Companies assign value to data by default when they decide that it is important to secure it. That is part of risk management - how much is it worth in dollars, reputation, etc. to have specific data stolen. However, there is also a huge amount of potential value in information that tends to go un-quantified. A lot of companies understand that they should use the data they produce to do something, but they have a difficult time building the business case and the large-scale implementation plan required to achieve what they need. This is where information governance helps. Though the initiative still requires IT support via the appropriate tools, information governance is much more business-centric. It focuses on data classification and metadata, data architecture and availability, data quality both via business and technical processes, data awareness and literacy, information life cycle management, stewardship, and value creation. As noted previously, there is some overlap with data governance, and there are certainly technical aspects typically handled by IT. However, the initiative is really about business colleagues helping to produce clean data by using their platforms correctly and then having access to that data with a high level of literacy and a sense of ownership so that they can ensure that it is of high quality and use it to create information, knowledge, and eventually value. Information governance is the framework that allows a company to assess their current state, create a plan for the future, and track progress towards the final goals. It is also a great basis for creating a business case for undertaking something like building a Data and Analytics program.
This leads to my second and third arguments for having separate data and information governance initiatives.
2) Information governance is often overlooked. Most companies have some sort of structured plan for addressing security, risk management, etc. around their data, but they frequently do not have a structured plan to address the actual use of that data to turn it into information and knowledge so that they can derive value. Splitting the initiatives ensures that ample focus can be applied to both.
3) There is a separation of duties required to focus on the appropriate aspects of governing data and governing information that can best be achieved by having IT run the data governance while folks on the business side run the information governance. Of course, there is some overlap and significant collaboration required to be successful.
Friday, October 18, 2019
Creating a Successful Data and Analytics Program - Continue to learn and refine (Part 6 of 6)
This is part 6 in a 6-part series about creating a successful Data and Analytics program. To read parts 1, 2, 3, 4, or 5, please follow the links below.
Step 5: Continue to learn and refine
As with any complex undertaking, one must accept that 100% success is essentially impossible the first time around. Even if you've done every step in a manner you believed to be correct at the time, things will inevitably go wrong, and that's OK. Initial execution should always be followed by a period of reflecting, learning, and refining. Perhaps the business needs are not exactly as planned. Perhaps priorities must change due to customer demands. Perhaps there are technical issues that require resolution. All of these are common situations that arise after the first implementation of a Data and Analytics initiative. The key is to have a flexible platform and open mind. As I stated in previous posts, iterating on an idea and continuous improvement are not an admission of failure nor an assignment of blame. This is exactly what is supposed to happen. Gone are the days of old-school business thinking where a company needed only to focus on doing a small set of things as perfectly as possible for as long as possible. Technology, business, and culture are changing far too rapidly to get yourself all worked up over achieving perfection.
Business now is in a constant state of evolution. There is a quote commonly attributed to Charles Darwin and his research on evolution that is actually just paraphrasing his writings but still captures the essence very well. "It is not the strongest species that survive, nor the most intelligent, but the ones most responsive to change." This concept can be directly applied to modern business. No matter how strong, smart, and high-performing a company is right now, it is the company's ability to change that will make it successful in the long run. Change does not just happen at the top when setting the direction for an entire company. It must be imbued into the culture so that even something as small as iterating on a Data and Analytics project is looked upon as a positive. As long as you have spelled out your ambitions and decided what success looks like, you will get a really good idea what is on-track and what is off-track from the metrics you've designed to monitor your progress. It may even be necessary to change the criteria for success and/or the metrics. Just be sure to proceed collaboratively, honestly, and transparently. Prioritization during this phase is key. Everybody will have an opinion, and there will be so many possible avenues to take that it is very important to approach them in an organized manner to keep things manageable. The methodology for prioritization will change from company to company, but it must always align with the success criteria.
Speaking of change, the tools and techniques in the world of Data and Analytics are changing very rapidly. In my experience, this change is even faster than most other technologies. Although it is not necessary for most companies to stay on the bleeding edge, it can be easy to fall behind your competitors or become stuck with older, incompatible versions of software if you do not stay on top of things. Keep this in mind as you build out your Data and Analytics program. You don't need to latch on to every shiny object, but you do need to stay educated on the latest trends and use your success criteria to decide which will help you get closer to that success state or even adjust your idea of success to set the bar even higher.
- Part 1 - Introduction
- Part 2 - Spell out your ambitions
- Part 3 - Select the organizational structure
- Part 4 - Perform a proof-of-concept
- Part 5 - Execute
Step 5: Continue to learn and refine
As with any complex undertaking, one must accept that 100% success is essentially impossible the first time around. Even if you've done every step in a manner you believed to be correct at the time, things will inevitably go wrong, and that's OK. Initial execution should always be followed by a period of reflecting, learning, and refining. Perhaps the business needs are not exactly as planned. Perhaps priorities must change due to customer demands. Perhaps there are technical issues that require resolution. All of these are common situations that arise after the first implementation of a Data and Analytics initiative. The key is to have a flexible platform and open mind. As I stated in previous posts, iterating on an idea and continuous improvement are not an admission of failure nor an assignment of blame. This is exactly what is supposed to happen. Gone are the days of old-school business thinking where a company needed only to focus on doing a small set of things as perfectly as possible for as long as possible. Technology, business, and culture are changing far too rapidly to get yourself all worked up over achieving perfection.
Business now is in a constant state of evolution. There is a quote commonly attributed to Charles Darwin and his research on evolution that is actually just paraphrasing his writings but still captures the essence very well. "It is not the strongest species that survive, nor the most intelligent, but the ones most responsive to change." This concept can be directly applied to modern business. No matter how strong, smart, and high-performing a company is right now, it is the company's ability to change that will make it successful in the long run. Change does not just happen at the top when setting the direction for an entire company. It must be imbued into the culture so that even something as small as iterating on a Data and Analytics project is looked upon as a positive. As long as you have spelled out your ambitions and decided what success looks like, you will get a really good idea what is on-track and what is off-track from the metrics you've designed to monitor your progress. It may even be necessary to change the criteria for success and/or the metrics. Just be sure to proceed collaboratively, honestly, and transparently. Prioritization during this phase is key. Everybody will have an opinion, and there will be so many possible avenues to take that it is very important to approach them in an organized manner to keep things manageable. The methodology for prioritization will change from company to company, but it must always align with the success criteria.
Speaking of change, the tools and techniques in the world of Data and Analytics are changing very rapidly. In my experience, this change is even faster than most other technologies. Although it is not necessary for most companies to stay on the bleeding edge, it can be easy to fall behind your competitors or become stuck with older, incompatible versions of software if you do not stay on top of things. Keep this in mind as you build out your Data and Analytics program. You don't need to latch on to every shiny object, but you do need to stay educated on the latest trends and use your success criteria to decide which will help you get closer to that success state or even adjust your idea of success to set the bar even higher.
Creating a Successful Data and Analytics Program - Execute (Part 5 of 6)
This is part 5 in a 6-part series about creating a successful Data and Analytics program. To read parts 1, 2, 3, or 4, please follow the links below.
Step 4: Execute
Step 4.1: Assemble the necessities
Assuming the items identified in the prior parts of this series have been satisfied, there are 3 necessities when implementing a Data and Analytics program.
First and foremost is the data. Typically, having data is not an issue for most companies. In fact, there is often too much, and that is one of the factors that instigates the need for a Data and Analytics initiative in the first place. However, it is very important to find and understand the data and the business processes behind its creation. A common term you will hear in conjunction with this step is, “source of truth”. This refers to the fact that every single piece of data can be tracked back to the system in which it was originally created. The source of truth should be found for as much data as possible which should then be collected into a central repository in its completely raw, unmodified form. This is the foundation for all downstream analytics, reporting, and other tasks, and gathering the data in this manner ensures the ultimate flexibility in its use.
The second necessity is the platform. To house and work with the data that is collected both initially and on a continuing basis requires a powerful, flexible system. It must be able to scale up and down in terms of both storage and computing power as needs arise while serving multiple use cases as outlined previously in part 4 of this series.
The third and final necessity is the expertise. Without expertise, either in-house or external, the initiative will never take off. There are very specific skillsets, like that of a Data Scientist, Data Engineer, or Data Analyst, that are required to both implement and utilize the platform. Whenever possible, core business and data knowledge should be learned by in-house resources while tasks like configuring and maintaining the underlying infrastructure can be outsourced with minimal difficulty. Due to the popularity of Data and Analytics, skilled local, near-shore, and off-shore external resources are readily available and should be utilized appropriately to ensure the most cost-effective implementation in combination with in-house colleagues.
Step 4.2: Horizontally expand analytics capabilities
Note: In this section, we are assuming the implementation of a Center of Excellence (CoE) model as it is generally the model that I prefer and could work well in a majority of situations. Horizontal expansion applies in all cases but would proceed somewhat differently for each. For instance, step 1 below would be important for a fully centralized, but 2 and 3 would be less so. Though, it is still always important to have the users who will benefit from the final product highly involved in its creation. Much like getting kids to eat a proper dinner, people are more likely to use something they had a hand in creating.
When we think about horizontal expansion, perhaps one might picture their waistline around the holidays. In this instance, though, horizontal expansion refers to the introduction and adoption of analytical skills and data literacy across a wide range of the business – well beyond the confines of a CoE. This expansion of knowledge and skills involves members of IT and most facets of the business with everything done under the guidance and supervision of the CoE.
There are three levels of Data and Analytics knowledge and use that apply to this expansion. I've listed them here in order of increasing complexity.
- Part 1 - Introduction
- Part 2 - Spell out your ambitions
- Part 3 - Select the organizational structure
- Part 4 - Perform a proof-of-concept
Step 4: Execute
Step 4.1: Assemble the necessities
Assuming the items identified in the prior parts of this series have been satisfied, there are 3 necessities when implementing a Data and Analytics program.
First and foremost is the data. Typically, having data is not an issue for most companies. In fact, there is often too much, and that is one of the factors that instigates the need for a Data and Analytics initiative in the first place. However, it is very important to find and understand the data and the business processes behind its creation. A common term you will hear in conjunction with this step is, “source of truth”. This refers to the fact that every single piece of data can be tracked back to the system in which it was originally created. The source of truth should be found for as much data as possible which should then be collected into a central repository in its completely raw, unmodified form. This is the foundation for all downstream analytics, reporting, and other tasks, and gathering the data in this manner ensures the ultimate flexibility in its use.
The second necessity is the platform. To house and work with the data that is collected both initially and on a continuing basis requires a powerful, flexible system. It must be able to scale up and down in terms of both storage and computing power as needs arise while serving multiple use cases as outlined previously in part 4 of this series.
The third and final necessity is the expertise. Without expertise, either in-house or external, the initiative will never take off. There are very specific skillsets, like that of a Data Scientist, Data Engineer, or Data Analyst, that are required to both implement and utilize the platform. Whenever possible, core business and data knowledge should be learned by in-house resources while tasks like configuring and maintaining the underlying infrastructure can be outsourced with minimal difficulty. Due to the popularity of Data and Analytics, skilled local, near-shore, and off-shore external resources are readily available and should be utilized appropriately to ensure the most cost-effective implementation in combination with in-house colleagues.
Step 4.2: Horizontally expand analytics capabilities
Note: In this section, we are assuming the implementation of a Center of Excellence (CoE) model as it is generally the model that I prefer and could work well in a majority of situations. Horizontal expansion applies in all cases but would proceed somewhat differently for each. For instance, step 1 below would be important for a fully centralized, but 2 and 3 would be less so. Though, it is still always important to have the users who will benefit from the final product highly involved in its creation. Much like getting kids to eat a proper dinner, people are more likely to use something they had a hand in creating.
When we think about horizontal expansion, perhaps one might picture their waistline around the holidays. In this instance, though, horizontal expansion refers to the introduction and adoption of analytical skills and data literacy across a wide range of the business – well beyond the confines of a CoE. This expansion of knowledge and skills involves members of IT and most facets of the business with everything done under the guidance and supervision of the CoE.
There are three levels of Data and Analytics knowledge and use that apply to this expansion. I've listed them here in order of increasing complexity.
- Consume and take action – This capability applies to just about every colleague from senior leaders to font-line workers. All the data, analytics, reporting, forecasting models, and prescriptive suggestions are meaningful only if colleagues crafting the business plans and executing the day-to-day tasks learn to consume the reports, dashboards, and forecasts, trust them, and use them to influence their decisions.
- Discover and share – A smaller number of colleagues will have access to modern tools like Microsoft Power BI, Tableau, etc. to discover data, analyze and create visualizations with that data, and share those visualizations with their colleagues. The CoE ensures that the tools, clean and organized data, best practices, and training are available to empower colleagues to help themselves via this top tier of self-service. The CoE can also certify what has been built to validate that they are using trusted sources of data and appropriate business logic as part of a larger Data Governance initiative.
- Strategize and wrangle – The volume and complexity of data available to an average business today is generally more than could ever be handled by a single team. Thus, it is critical that a set of Data Champions spread throughout the organization have tools and abilities to contribute to this effort. “Strategize” refers to the act of coming up with the list of key data sources, associating those with business processes, and building and prioritizing the use cases that will best serve their area of the business. These Data Champions take ownership for the planning and delivery of these use cases even if they are not responsible for execution. The work may fall back to the CoE, an external vendor, or other colleagues, but the business must at least own the requirements and the eventual usage. “Wrangling” is a somewhat recent term in the Data and Analytics realm coined to describe the act of gathering, understanding, cleaning, and organizing data. Modern tools like Informatica, Trifacta, Paxata, etc. provide interfaces for Excel-jockeys to work in a familiar environment but with potentially terabytes or petabytes of data behind the scenes. It is important to remember that wrangling is a significant undertaking that can consume 50% of all time spent on any given use case. This effort also involves the Data Champions taking responsibility as stewards of the data that falls in their area of the business. Similar to bullet 2 above, the output of wrangling can be certified and operationalized by the CoE as part of a Data Governance initiative.
Thursday, October 17, 2019
Creating a Successful Data and Analytics Program - Perform a proof-of-concept (Part 4 of 6)
This is part 4 in a 6-part series about creating a successful Data and Analytics program. To read parts 1, 2, or 3, please follow the links below.
Step 3: Perform a proof-of-concept
As with most new endeavors, it is difficult to dive head-first into Data and Analytics without first implementing a scale experiment to ensure you have the correct people, technology, leadership support, and business interest. That said, a lot of companies dip their toes into analytics, find minimal value, and then slow down or back away from the initiative. In some cases, this could be a perfectly appropriate response. However, it is sometimes a reaction to a poorly conceived proof-of-concept (PoC). When planning the use case and scope for your Data and Analytics PoC, be sure to consider the following.
- Part 1 - Introduction
- Part 2 - Spell out your ambitions
- Part 3 - Select the organizational structure
Step 3: Perform a proof-of-concept
As with most new endeavors, it is difficult to dive head-first into Data and Analytics without first implementing a scale experiment to ensure you have the correct people, technology, leadership support, and business interest. That said, a lot of companies dip their toes into analytics, find minimal value, and then slow down or back away from the initiative. In some cases, this could be a perfectly appropriate response. However, it is sometimes a reaction to a poorly conceived proof-of-concept (PoC). When planning the use case and scope for your Data and Analytics PoC, be sure to consider the following.
- Spell out your goals, document what success looks like, and develop a methodology to measure and report on progress towards that success state. Things can evolve along the way, but you need a foundation upon which to build so you are not just throwing things at the wall to see what sticks.
- Select an appropriately sized use case such that it can be completed in a reasonable amount of time. You want something significant enough to provide useful output but small enough that it is achievable in the time allotted. That time can vary, but 3 months is not unreasonable to implement something new and fairly complex with a high potential value.
- Take the opportunity to build support by selecting a use case of interest to business leadership instead of focusing solely on the technology and technical leadership. The more colleagues who have a stake in the success of the PoC, the more likely they are to contribute expertise, and the more likely the PoC is to be successful. Do not implement the PoC in secret and then spring the results on the business in an I just figured out how to do your job better approach.
- If return on investment (RoI) is an important consideration for your Data and Analytics program, select a use case with notable, calculable time saving or monetary saving/earning numbers. This is not to say that the PoC is going to directly save time or save/make money, but you should be able to clearly show how it could lead to that when put into production.
- Analyze your company's current state of data readiness and scope the use case appropriately. This can be a tough one. If your company already has a solid platform in place to handle day-to-day operational reporting and basic analytics needs, then the PoC should demonstrate the potential to do more with advanced analytics like machine learning. If your company is still struggling with the basics, a PoC that overshoots the current state by too much may not be the right choice. Such a PoC would tend to have one of two impacts. First, it could result in the question of why would the company invest in something so far-reaching before it solves for daily needs. Second, it could get colleagues and leadership excited about a future they think is just around the corner when it is actually years away and get you into an over-promise and under-deliver situation. That said, you do want to engender a degree of excitement to garner support. So, finding a balance is key.
- If you do opt for a PoC focused on forecasting, machine learning, or other advanced analytics, be sure to gather enough data to actually find actionable insights. There may be hundreds or even thousands of variables to consider before you can pare things down to a reasonable size and still provides business value. Do not go into the PoC thinking that you are going to build a machine learning model on a couple dozen variables. You are likely to be disappointed. Also, be prepared for the possibility that your first endeavor may not provide significant insights or it may just confirm commonly held beliefs. What you learn through the process is still important, and you can also pivot to something more promising. Communication and transparency is key.
- Do not miss the forest for the trees by seeing the platform as only capable of serving a single or limited set of purposes. Step back and consider the possibilities. A correctly architected Data and Analytics platform can do all manner of things. I have listed a few below, and it is important to figure out what will best speak to the leadership of your company.
- Analytical insights – Use the data to find patterns, forecast the future, and prescribe actions with the highest potential success rates.
- Process offloading – Use the massive compute power and inexpensive storage inherent to the platform to offload long-running jobs currently performed by other platforms.
- Active data archiving – Store all data, forever, on a platform where storage is cheap and the data is easily and securely available for analytics and reporting
- Real-time alerting - Process data in near real-time and produce proactive alerts when a non-conformance is found or a specific state is reached.
- Self-service – The biggest current trend in Data and Analytics is the democratization of capabilities so that users are empowered to help themselves instead of waiting in line for their needs to be prioritized and addressed
- If at all possible, use platform (PaaS) and software (SaaS) service offerings available from cloud providers like Amazon, Google, IBM, Microsoft, etc. This is particularly true in a PoC environment that you want to build out rapidly and then dispose of or possibly use for future experimentation. However, the cloud also provides significant benefits when moving to the production phase. People greatly underestimate just how difficult it is to build, maintain, and tune something like a Hadoop cluster either on premises or in cloud infrastructure (IaaS). It is extremely time-consuming and can be very expensive. Cloud services allow you to focus on satisfying business needs instead of fighting with the technology while being easy to expand as demands increase. Done correctly, the cloud can also be significantly less expensive. If your company does not currently allow it, find some partners in the organization, build a business case, and fight for it as the benefits are huge.
Wednesday, October 16, 2019
Speed Versus Structure In Information Technology
Since the explosion of Information Technology, it has become a necessity for companies of all types and sizes. From a small start-up to a mega-corporation, practically everybody wants to move fast and develop solutions with minimal red tape. There is a constant struggle to balance speed versus structure in most IT departments. I have worked on both ends of the spectrum. On one hand, I've entered companies who believed that they had a traditional, Waterfall methodology when what they actually had was chaos. On the other hand, I have done consulting with the most strict Agile you can find where the client, if dissatisfied with the work completed during the most recent 2-week sprint, would not pay for those two weeks. What I have discovered over the last 20+ years, though, is that there is no silver bullet. There is no single approach that will work for every situation. However, I can guarantee that there is a specific level of structure in every situation that will optimize the team and what they produce. You just need to analyze the current status and introduce structure as-appropriate while not being afraid to change and iterate until you find that sweet spot. I will provide a few of my own scenarios below in the hopes that they can help you figure out what is best for your scenario.
Scenario 1: Hyper-strict Agile
The hyper-strict Agile scenario I noted above actually worked incredibly well. The teams I oversaw pumped out an impressive amount of very high quality code in those 2-week sprints, and our clients were very satisfied. There were risks, though.
- It took a very special person with a somewhat unique personality and work ethic to excel under those conditions.
- Colleagues who did not fit dropped out very quickly, but those on the borderline who maybe could have excelled if managed better had a high level of stress-induced burn-out.
I found that there were 2 keys to success in this scenario.
- Treat all colleagues fairly, honestly, and openly while ensuring that they play a pivotal role in selecting the work to be completed each sprint. Certainly, part of the leader's job is to set high expectations, but having a team fully committed to the work before them instead of just trying to meet expectations allows for great things.
- Lead from the front, not from the back. If you have a team you expect to work hard and put in long hours, then you must be willing to do the same. In my situation, I intentionally did not self-assign tasks in each sprint. However, I made it my mission to provide an extreme level of assistance to any colleagues falling behind and, in the second week of each sprint, to pick up every dropped task and complete them myself or in collaboration with the assigned developer. This had several positive ramifications. The first was engendering a real team spirit when they saw that the person leading them was also in the trenches with them. Equally as important, though, it gave me the opportunity to truly understand each colleague on the team, their capabilities and interests, where they excelled, and where they had opportunity for improvement. This allowed us to tweak assignments each sprint to move even faster while still maintaining quality and keeping the team happy and learning new things.
Scenario 2: Waterfall
In both start-up companies with which I have been involved, we generally followed a Waterfall methodology with what I would call a touch of Agile. Primarily, the Agile portion was that we had one meeting a day to discuss progress, roadblocks, and plans. These scenarios were a bit unique, though, and the keys to success might not be achievable in other situations.
- We were a small group of people all extremely committed to a single goal.
- All team members were highly self-motivated and entrepreneurial. It was ingrained into their personalities that they were going to work hard and produce the best output they could. Plus, it didn't hurt that every task they performed essentially fed directly into their own personal success or failure due to their tight alignment with and ownership stake in the company.
- Any addition to the team would quickly fit in per the previous 2 bullets or be replaced as we didn't worry about things like HR policies.
This has been my personal experience with highly-successful Waterfall, but I'm sure there are other situations where it can work. The key is that a lot of companies seem to refer to anything that isn't Agile as Waterfall. Per the next scenario, what they actually often have is chaos.
Scenario 3: Chaos
Just the word chaos generally has negative connotations, and I would certainly argue that chaos is a strong negative when it comes to Information Technology. I walked into a new job a few years back and instantly recognized the telltale signs of chaos. Prior managers who had since left used to micro-manage and dole out tasks to individual team members while dissuading collaboration. Daily processes were constantly running long and/or failing. Only through the extreme efforts of key individuals on the team was everything just barely holding together. I interviewed every member of my new, large team and realized that they all had a decent idea of their own roles and the work they were doing, but had almost no idea what was going on with the rest of the team. They also all confirmed that there was clearly no process for work intake, prioritization, testing, releasing, and teamwork. In this scenario, we very quickly started to introduce some order into the chaos through structure. Certainly their were doubters both within and external to the team, but we took the time-tested approach of clearly stating our goals and developing a methodology to measure and report on our progress towards them. It is difficult for doubters in such a situation to argue with hard metrics. It did not happen overnight, be we were able to, without expanding the team, increase throughput 3-fold, improve our rate of achieving SLAs by 500%, and shrink our weekly pending tasks list (essentially bugs and non-project items) by 95% into the single digits. It took 5 months to achieve these numbers, but the initial benefits showed up very quickly. A tremendous side-effect was the overall happiness and satisfaction of the team. People even started doing their week-long on-call rotations without a single incident so they could sleep through the night which was better for everybody. All of that was made possible by finding the right amount of structure for the situation. We continued tweaking and adjusting that structure for a year through both our team organization, our procedures, and our tools (e.g. Jira) until we felt like it was just about as good as it could get.
I hope that maybe some of the information I have shared above can help you improve your own scenarios. Continuous improvement and a desire to implement change is neither an admission of failure nor an assignment of blame. It is simply a way to always be moving forward and making progress.
Creating a Successful Data and Analytics Program - Select the organizational structure (Part 3 of 6)
This is part 3 in a 6-part series about creating a successful Data and Analytics program. To read parts 1 or 2, please follow the links below.
Step 2: Select the organizational structure
It is obvious from the outset that a business must create some type of organizational structure around a Data and Analytics initiative as it requires the full-time effort of at least a small group of colleagues. There are four choices when selecting which organizational structure is right for your company, and all of them can lead to successful outcomes.
In part 4 of the series, we will perform a proof-of-concept...
- Part 1 - Introduction
- Part 2 - Spell out your ambitions
Step 2: Select the organizational structure
It is obvious from the outset that a business must create some type of organizational structure around a Data and Analytics initiative as it requires the full-time effort of at least a small group of colleagues. There are four choices when selecting which organizational structure is right for your company, and all of them can lead to successful outcomes.
- Business unit led - When business units have distinct data sets and scale isn’t an issue, each business unit can make its own Data and Analytics decisions with limited coordination. This approach is very flexible but will typically lead to independent silos of data. This may or may not be an issue depending upon the structure of the business units within the company and their need or desire to share data. It is possible that data independence is a key driver to success. This approach can also be more expensive if multiple teams are building their own programs. That said, the cost can be minimized by at least having a central technical platform for everybody to share. Thus, it would just be the "business" portion of the program that would differ across units.
- Business unit led with central support - Business units make their own decisions but collaborate on selected initiatives. This is a slight modification to the first approach where the business units occasionally come together on specific use cases. This collaboration is a good way to standardize certain data. Let us say, for example, that multiple business units accept customer complaints and record them. There is benefit to standardizing the codes for the set of possible complaints as well as the business processes for accepting and recording said complaints so that the metrics can be compared and contrasted across the business units. Because there is no full-time, central team identifying opportunities for and governing these collaborations, corporate leadership as well as the leaders of the various business units need to be committed to this collaboration. It is easy to miss an opportunity or for business units to diverge again over time.
- Center of Excellence - An independent center oversees the company’s Data and Analytics while business units pursue initiatives under the CoE’s guidance and coordination. This is the middle-of-the-road approach where a smaller, central team takes on the responsibility of identifying opportunities for collaboration and governing the use cases, but the business units are also allowed a significant degree of freedom through self-service capabilities. The CoE must truly be empowered by corporate leadership to serve in this governance capacity, and they also need the right approach and mindset to build a grass-roots movement of collaboration and self-service in addition to the top-down mandate.
- Fully centralized - The corporate center takes direct responsibility for identifying, prioritizing, and implementing all initiatives. In this case, business units are still involved but primarily to consult as subject matter experts on the data they use and their business processes. This approach allows the company to impose the highest level of governance, but it can be difficult and expensive to build a single team large enough to satisfy all of the data needs without becoming a bottleneck. Caution must also be taken to ensure that pursuing a "single source of truth" does not overshadow the actual, real-world needs of the business units. The program should always allow for some degree of governed flexibility.
In part 4 of the series, we will perform a proof-of-concept...
Tuesday, October 15, 2019
Creating a Successful Data and Analytics Program - Spell out your ambitions (Part 2 of 6)
This is part 2 in a 6-part series about creating a successful Data and Analytics program. To read part 1, please follow the link below.
Step 1: Spell out your ambitions
- Part 1 - Introduction
And now on to part 2...
Step 1: Spell out your ambitions
What will Data and Analytics provide that we cannot
already do? What do we expect the ROI be in terms of cost/time savings or
additional revenue? Where do we stand compared to our rivals? Will Data and Analytics provide us a strategic advantage in the marketplace or is it a
requirement just to stay relevant? These questions and more are critical to
consider as you embark on the Data and Analytics journey, and they need to be
answered to an extent that gains significant buy-in from senior leaders. Stepping back a bit, you are, at the very least, looking to understand why you are going on this journey, what success looks like, and how you will measure your progress towards success. This applies to all significant investments a company makes, but becoming data-driven is typically such a difficult undertaking (and often expensive) that having this framework in place is even more vital.
Note: Upper management support is not simply about money and prioritization. They, too, need to adopt Data and Analytics as part of their culture. If they expect a Manager or Director to run their operations according to specific KPIs, then the senior leadership needs to pay attention to those same KPIs and use them to review performance. If they expect others to accept and react to information gleaned from data, they have to be open to that as well even if the information runs contrary to long-held beliefs. If they want data literacy to expand beyond the Data and Analytics team, then they need to become data literate themselves and ensure that all teams are playing their role in the cultural evolution. HR, in particular, can be key to this evolution. At any medium to large company, it is not possible for a single team to achieve all of the change that is required. Significant support is necessary from leadership in all areas.
Along with answering questions like those posed above, a decision must be made on priorities. There are essentially four areas where
Data and Analytics can be utilized:
- Improve existing products and service offerings
- Build new products and service offerings
- Automate and optimize internal processes
- Transform business models
Done correctly, a business can scale out their Data and Analytics initiative to cover all four bullets, but it is important to start
with a narrower focus to avoid becoming overwhelmed and diluting the benefits
of the initial undertaking. The fourth bullet is also, typically, a much more significant endeavor than the other three and likely not where you would want to start unless it is a necessity.
In part 3 of the series, we will select our organizational structure...
In part 3 of the series, we will select our organizational structure...
Creating a Successful Data and Analytics Program - Introduction (Part 1 of 6)
This blog post is the first in a 6-part series that outlines a framework for implementing a Data and Analytics Program. The intention is not to delve into the
specific details of execution because there are many options that can work but to spell out the foundation and key
principles upon which a successful program is built. The information presented is an amalgam of details culled from multiple years of
research, experience, and conversations with others who have undertaken such an
endeavor. The guidelines and advice are based upon industry standards and best
practices along with lessons learned and the latest emerging technologies and
techniques.
In part 2 of the series, we will start by spelling out our ambitions...
Back to Blogging
After working for the last several years either in a start-up with 100 hour work weeks or for a couple of companies with significant restrictions on their employees' public image and information sharing, I am once again in a position to get back to blogging. Though I know this blog isn't particularly popular, it is a great creative outlet for myself and allows me to really align my thoughts. If something I write can also help at least one other person in their career or with a personal technical challenge, then I am very happy for it.
Subscribe to:
Posts (Atom)