How Leaders Manage AI Projects?

Marc-Etienne Dartus
7 min readFeb 2, 2022
Photo by Austin Distel on Unsplash

I. AI Project Management in Research Papers

Managing Artificial Intelligence projects is not always easy, especially when companies have never done it before. Several theoretical frameworks have been produced to identify the key stages of a project.

According to Nguyen-Duc and Abrahamsson (2020), the Lean Startup approach is appropriate. It allows us to understand the business objectives, transform them into requirements, and then into products. They propose to repeat this procedure three times with a very simple product as the first step, pretending to be an AI. This technique allows adapting the business needs and expectations without having to spend too much time during its development. Then, the next two steps will be to realize the product with the training data and then with real data.

This approach, although very interesting, may seem too vague and unstructured for some companies. According to Megler (2019), 4-point project management is proposed.

1. Create a Balanced Team

First, it is necessary to build an adequate team that can manage the whole spectrum of the project from the business part to the technical part. Several profiles are needed such as data scientists, data engineers, developers, users, and business representatives. Unfortunately, the “users are sometimes Product Owner which can bias the understanding of the customer compared to the Lean Startup method. In addition, most small and medium-sized companies cannot recruit as many people for a project, which is why according to Ransbotham et al. (2017) and Canhoto and Clear (2020) many turn to outsourcing.

2. Define The Expected ROI

Megler proposes to assess the economic value of the project. To do this, the primary indicator for this analysis is the return on investment (ROI). This indicator also makes it possible to prioritize projects and request funding. The great advantage of this measure is that it is understandable by any decision-maker because its value removes all the technical aspect that is induced with AI. However, this must be taken with caution, because the theoretical ROI determined at the beginning of the project is often quite far from the real one.

3. Establish a Risk Management Dashboard

AI being a subject with many uncertainties, it is necessary to list all possible hazards, define their impacts, and the actions that can be taken to avoid them. It will then be mandatory to update this document periodically to evaluate if new threats appear or if old ones change.

4. Reconsidering the Project

Lastly, Megler invites to reconsider the project at each of its major phases with the business initiators. The purpose of this reflection is, on the one hand, to state the progress of the project and to present the way in which the data is recovered, transformed, and used. On the other hand, it allows to update and discuss the risk management and define the next steps. In the end, this meeting should lead to one of four decisions:

  • To invest more
  • To continue as it is
  • To focus the project on another topic
  • To stop it

This last step will especially address the different expectations of project management and investment that are sticking points according to Ransbotham et al. (2017). The 4 steps described here are all the more important because they are presented by a data science expert at one of the leaders in artificial intelligence that is Amazon.

II. Structuring AI Project Management

To understand how value within companies is created, it is necessary to study how AI projects are managed. It appears that several key steps are necessary for this process.

1. Opportunity Identification

First, it is essential to define the need or opportunity that the project will help to address. Most people consider discussion with the business as the most efficient way to achieve this task, however other solutions exist such as those based on opportunities known to be solved by AI such as fraud detection or client churn.

2. Pre-project Reflection

Then, there is a whole phase of reflection prior to the project. Indeed, some needs do not necessarily require AI. Because of its high cost in infrastructure and qualified personnel, it is important to see if the problem cannot be solved without it. In addition, having a reflection on the digital maturity of the company is essential. This allows identifying the existing infrastructures and the skills present within the development teams. Finally, it is essential to have a reflection on the quantity, availability, and quality of data.

3. Definition of ROI and KPIs

Then, it is strongly advised to define the indicators related to the projects. Indeed, the ROI is a good indicator for any decision-maker to understand the issues and possible returns. However, it is sometimes complicated to estimate it, this is why business-related KPIs are determined to overcome this problem. Most leaders consider having this indicator of the success before starting the project.

4. Development

After having verified the interest of the project and defined the indicators, the development can begin. The vast majority of people advise using an agile methodology with short iterations. Some think that already making an AI-based solution is a good step from the first iteration, others recommend creating a solution based on simple rules.

To do a rule based algortihm, first you need to understand in detail the parameters of the problem. A tips is solving manually the first treatments to fully understand the project issues. These tasks will make it easier to define naive rules for the proof of concept of the algorithm. Then, on the next iteration, people advise creating a first basic AI. Indirectly with the implementation of these first iterations, data will be generated and can then be used to perfect the algorithm in the next iterations.

5. Iterative Review

Finally, it is through this short iterative process that it is possible to verify at each step that the project corresponds to the business need. If this is not the case, at the end of each cycle, actions can be taken accordingly. However, if it turns out that for some reason it is necessary to stop the project, it is important to study the causes of this situation. Understanding why a project failed is essential to avoid making the same mistakes later.

However, as some people have pointed out, the different stages of project management do not really differ from an innovation project. Indeed, only the context of the exchanges and reflections within the different parties is particular to the AI domain. As for the structure of the project, it corresponds to the one proposed by the Lean Startup approach.

III. Reflection during project iterations

1. Iterative Review

At each iteration of the project, a very important phase is to verify that the project corresponds to the needs and criteria defined beforehand. Several approaches can be taken depending on the state of the project. On the one hand, if the project is on track, it is possible to continue without making any changes or to invest more if necessary. On the other hand, if the project is not meeting expectations, one option may be to stop it. However, it is wise to consider the possibilities of reframing the project if the need is not appropriate or to reduce the scope of the different tasks to be performed.

2. Post Mortem Study

If the project fails, it is important to identify the source of the problem so that it does not happen again in the future. To do this, it is possible to check the status of the various criteria of the pre-project analysis. Indeed, a failure on the objectives and capacities of the company can have repercussions on the viability of the project. Alternatively, the error may be due to other blocking factors related to the realization of the project.

However, even if the project is abandoned, it is important to identify the potential that has already been developed to not lose the entire investment. The goal is to reuse certain parts that are sources of independent value. Unfortunately, this retrospective is often neglected in companies. It is often difficult to have a critical look at one’s own project. Moreover, some people do not necessarily want to show the management problems within their teams and to highlight the various dysfunctions.

Conclusion

The major difference with the method proposed by Megler (2019) and Nguyen-Duc and Abrahamsson (2020) is that there is no assessment of the risks blocking the successful achievement of the goal. However, the proposed approach proceeds to remove these risks from the beginning with the various pre-project reflections.

However, the aspect of evaluating the cause of project failure is not addressed in the research paper. Nevertheless, part of Megler’s method is based on project risk analysis. A combination of the two would be suitable to manage the risk assessment throughout the project in the best possible way, thus reducing the probability of its failure.

Canhoto, A. I., & Clear, F. (2020). Artificial intelligence and machine learning as business tools : A framework for diagnosing value destruction potential. Business Horizons, 63(2), 183‑193. https://doi.org/10.1016/j.bushor.2019.11.003

Megler, V. M. (2019). Managing Machine Learning Projects Balance Potential with the Need for Guardrails. Amazon White Paper, February.

Nguyen-Duc, A., & Abrahamsson, P. (2020). Continuous experimentation on artificial intelligence software : A research agenda. Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 1513‑1516. https://doi.org/10.1145/3368089.3417039

Ransbotham, S., Kiron, D., Gerbert, Ph., & Reeves, M. (2017). Reshaping Business With Artificial Intelligence : Closing the Gap Between Ambition and Action. MIT Sloan Mangement Review and The Boston Consulting Group, 59(1), 1‑17.

--

--

Marc-Etienne Dartus

🚀Software Engineering at @Amazon Madrid. 🤖 Specialized in Data Science. 👨🏻‍💻 https://github.com/medartus. Marketing Newsletter: https://www.foundernotes.io