Aussie AI

Chapter 9. Budgeting and ROI for an AI Project

  • Book Excerpt from "Generative AI Applications: Planning, Design and Implementation"
  • by David Spuler

Chapter 9. Budgeting and ROI for an AI Project

Budget Allocation

It’s time to work out your budget allocation for the new generative AI project. Here’s the modern way:

        You: I might need some staff for my gen AI project...

        CEO: We have 2,000 developers on staff, but you can only have 1,999.

        You: Why not all of them?

        CEO: I need the WiFi fixed at my house.

All joking aside, deciding on the budget allocation for your generative AI projects is important. The prominence of generative AI in the marketplace has convinced the C-suite executives to provide new funding for LLMs and their ilk. Another common aspect is that funding is being diverted from other IT projects:

  • Classic ML projects
  • Non-AI SaaS software projects

This is an important decision to make, and not without risk. The payback from generative AI projects is not always clear, whereas ML and SaaS have been good earners. I expect that as the generative AI’s shine comes off a little bit, there will be a reversion to some of these software technologies for their solid ROI.

As an example data point, Meta’s recent earnings call showcased a lot of top-line revenue growth from AI, but it seemed mostly attributable to ML projects, rather than generative AI, despite the many billions they’re spending on training very advanced LLMs. Another data point is Microsoft’s CFO characterizing their GPU purchases as a 15-year investment.

One important tip if you’re looking at sorting out the overall budget for an AI project: don’t use generative AI! Although they’re great with words, LLMs are well-known to be rather doubtful when it comes to numbers. AI is not going to do well at adding up the columns in your spreadsheet. I don’t think Microsoft Excel is in any danger in the near future. In fact, I expect AI is likely to call out to Excel in the future, using it to sum columns and slice-and-dice the data into 12 different charts, which the LLM can then explain using poetry.

AI Budget Items

Budgeting for tech projects is surely something you are already familiar with. An AI project has a lot of similarity with any other non-AI project, so I’m only going to discuss some of the nuances that arise with AI.

Some of the main up-front project costs are likely to be:

  • Consultancy and advisory costs
  • Legal fees
  • Capex for servers and GPUs (if self-hosting)
  • Staff re-training costs
  • New AI staffing costs
  • Data cleaning costs
  • Model training and fine-tuning costs
  • New development costs

To the last point about extra development costs, the AI platform APIs tend to offer “usage-based pricing” which is a different pricing model from most commercial SaaS software platforms. Pay by the token does not care if it’s a developer or customer using the LLM. In the past it wasn’t the norm to pay extra licensing fees to a software vendor whenever your QA department was testing your IT project.

Some of the launch costs when you are just about to go live include:

  • Final integration and beta testing
  • Launch costs (general costs of going live)
  • Promotional launch costs (internal or external)
  • Post-launch update readiness (e.g., staff ready to fix safety/toxicity queries).
  • Scaling costs (if it booms)

Some of the additional add-on technical projects that are specific to AI include:

  • Model evaluation costs (testing fine-tuned models)
  • Safety evaluation tools
  • RAG architecture components (e.g., datastore, retriever)

The ongoing costs after going live with the app are likely to be:

  • Cloud hosting fees (server hosting)
  • LLM token fees or GPU-specific hosting fees
  • Staff costs
  • AI-specific performance accelerators (buy or build)
  • Safety-related components (e.g., prompt shields)
  • Fine-tuning or “per-tenant” training costs

Hopefully, there’s some money left after all this, so you can make payroll this week. You know what would be nice? Having an AI project bring some money back in.

ROI

The ROI of an AI project goes like this: you spend $50 billion dollars, and in return you get back three or four. Sound good? That’s the AI industry’s overall figures so far.

The first question about ROI is to review your business goals. Your AI might be a “loss leader” for the next few years, or perhaps forever. Rumor has it that many big AI vendors offer their services for $20/month, even though it costs them $40/month. Are there any AI vendors that are profitable from LLM inference other than NVIDIA?

Pricing strategies are also changing over time, especially with the prominence of usage-based pricing for token creation. Hence, yet another factor to consider in planning is that the pricing trends are currently:

    (a) API-based LLM inference costs have been declining over the last year or two, and

    (b) GPU-specific vendor hosting costs are plummeting as well, and

    (c) Open-source LLMs and engines are holding their own in terms of capabilities (although OpenAI is still leading the “reasoning race”).

Hence, to get a good understanding of your future cost model, you need to plan for these trends, albeit with some difficulty in projecting them ahead multiple years. It may be that within six months to two years, the costs will be much further down, and ROI can be higher.

In terms of business AI projects, calculating the basic tangible ROI means you have to measure:

  • Cost savings
  • Revenue increases
  • Project cost

A lot of the cost issues are discussed above as budget items. There are also some unusual costs in the product development phases due to token-based pricing from LLM API vendors or hourly billing in GPU hosting rental costs.

However, a lot of the benefits of AI projects are expected in the longer-term. Some of these intangible benefit points include:

  • Increased staff productivity
  • Building a “core competency” in AI
  • Better customer communication (e.g., personalized, well-presented marketing copy)
  • Customer retention and opportunity for upsell.
  • Keeping up with competitors (e.g., every marketing department in the world is adding AI tech to their website, even if you don’t have it yet).
  • Gathering extra data for future training projects
  • Internal employee satisfaction and morale (automating mind-numbing tasks).
  • Development organization flexibility to quickly release new capabilities.
  • Overall operational efficiency gains (e.g., streamlined workflows)
  • Improved competitive advantages and protection from disruptors.
  • Building towards an internal “AI platform” for greater gains.

Cost calculations are the third factor in an ROI analysis. The overall costs should be grossed up to get a figure for the Total Cost of Ownership (TCO) for the entire AI project. The costs are mostly tangible, but there is also the opportunity cost of other lost projects to consider.

Staff Skills and Salaries

I have to say that staff salaries in AI are massive, if you’re a developer. If you’re an executive reading this who needs to hire some AI staff, here’s a little secret:

        You don’t need AI developers.

Those million-dollar salaries for AI developers that get mentioned in the press about OpenAI and other companies? Those are mostly illusory unless you happen to be one of the hundred or so people on the planet who know how to train a trillion-parameter foundation model.

But you don’t need to build one of those massive LLMs thanks to OPM: Other People’s Models. You can rent commercial models from OpenAI and many other vendors, or you can load up the open source models on your own hardware if you prefer. Either way, you don’t need to do any mega-size training projects.

What you probably most need is:

  • System administrators
  • Build engineers
  • Cloud engineers
  • Python developers
  • Data scientists

I can’t believe I’m writing this, but you might not even need a system-level programmer, because you don’t need to build your own AI engine. The main place that such expertise will be used in your architecture is inside the inference backend that runs your chosen LLM. Again, this coding is done by high-paid developers at commercial LLM platform startups or for peanuts by open source contributors. Nobody at your company even needs to look under the hood at all the code.

I suspect it will not be long that the term “programmer” will go away. You will just have a “technologist” who can do all the above tasks by running the AI to write the code, tweaking it and productizing it all. Someone will always be needed to fix the wifi.

Financial Optimizations

An AI project is expensive in terms of the hardware, the software, and the people you need. There are some considerations that can reduce the cost somewhat.

Use existing assets. What internal data assets do you possess? Can you re-purpose any of your company’s existing hardware assets? And can you “re-purpose” any of your staff, too?

Buy vs rent. If it’s floating, flying, or foundational modeling: rent, don’t buy! Similarly, do you need to buy your own servers and GPUs? The decision may be different for the different phases of a project:

  • Development and testing
  • Training the model (fine-tuning/specialization)
  • Inference (live execution)

For example, you might want to buy for training phases and rent for the inference phase. This depends on how much training you need, the size of your model, and whether you plan to avoid fine-tuning for proprietary data by using RAG instead. The cost of inference depends on the user counts, which is significantly different if it’s an internal employee project versus a live public user application.

Idle VMs and GPUs. Watch out for virtual machines and rented GPUs being idle early in the project. You’re paying money for nothing in such cases. This can occur in the development phases and in the early live deployment when user levels are low.

Scrimp on developer models. During the development and testing phases, there’s no need for gold-plated AI models. The cost of development and testing of your AI application can be reduced by using low-end models for simple testing. Many of the components needed are not dependent on whether the AI engine returns stellar results. Initial development, prototyping, and ongoing regression testing of these parts of the system can proceed with small models.

There is also vendor support for testing on lower-end models. There are various other AI platforms that offer interfaces that mimic OpenAI’s API, but at a lower cost, so you can test on these platforms, and then do final testing on the live commercial platform.

Technical Debt in AI Projects

Everything’s changing fast in AI research and industry practices. Hence, the current methods of building and deployment AI applications are a work-in-progress. Nobody really knows what’s optimal in regard to:

  • What to use AI for?
  • Which models?
  • What tech infrastructure?
  • How to optimize?
  • Safety concerns?

Hence, as part of planning an AI project, consider paying more attention to “technical debt” inherent in this situation. You may need to refresh your tech stack much sooner than in a non-AI project. It’s hard to quantify in terms of effort or timescales, but it’s an important issue to make note of in your AI project proposals. The key point is mainly to budget for extra funding for post-launch maintenance tasks.

References

  1. Peter Cohan, Sep 23, 2023, Why Companies Buy Generative AI Consulting: The 3-Month Payback Factor, https://www.forbes.com/sites/petercohan/2023/09/23/why-companies-buy-generative-ai-consulting-the-3-month-payback-factor/ (AI projects with a 3-month payback ROI.)
  2. Kaya Ginsky June 27, 2024, Figma CEO says it is ‘eating cost’ of AI upgrade for customers in 2024, https://www.cnbc.com/2024/06/27/figma-ceo-says-its-eating-cost-of-ai-for-customers-in-2024-upgrade-.html
  3. Pecan AI Team, April 4, 2024, How to Measure (and Increase) the ROI of AI Initiatives, https://www.pecan.ai/blog/how-to-measure-increase-roi-of-ai/
  4. Mark Gurman, June 11, 2024, Apple’s Push to Infuse Devices With AI Will Take Years to Pay Off, https://www.bloomberg.com/news/newsletters/2024-06-11/will-apple-intelligence-features-boost-iphone-sales-it-may-take-years
  5. CNBC, Aug 9 2024, The gap between AI expectations and outcomes in the workplace are wide, https://www.cnbc.com/2024/08/09/the-gap-between-ai-expectations-and-outcomes-in-the-workplace-are-wide.html
  6. Lucas Mearian, 22 Aug 2024, Generative AI is sliding into the ‘trough of disillusionment’, https://www.computerworld.com/article/3489912/generative-ai-is-sliding-into-the-trough-of-disillusionment.html
  7. Grant Gross, 10 Oct 2024, When is the right time to dump an AI project? https://www.cio.com/article/3555331/when-is-the-right-time-to-dump-an-ai-project.html
  8. Matt Harney, Oct 24, 2024, SaaSletter - Positive B2B AI Signs: Via Morgan Stanley + Cloud Ratings B2B AI Interest Index, https://www.saasletter.com/p/positive-b2b-ai-signs-october-2024

 

• Online: Table of Contents

• PDF: Free PDF book download

• Buy: Generative AI Applications: Planning, Design and Implementation

Generative AI in C++ The new Generative AI Applications book by Aussie AI co-founders:
  • Deciding on your AI project
  • Planning for success and safety
  • Designs and LLM architectures
  • Expediting development
  • Implementation and deployment

Get your copy from Amazon: Generative AI Applications