Chapter 4. Deciding on an AI Project

Chapter 4. Deciding on an AI Project

Decisions, Decisions

Come on, you don’t need to decide! It’s AI, and it’s already been decided for you. But there are a few other decisions:

• What AI projects to build?
• External versus internal projects?
• How soon do we need it?
• How many users to support?

And then there are some meta-decisions:

• Who decides?
• Who’s going to research the technology?
• Whose responsibility is building it?
• Who’s tracking the project?
• Who’s paying for it?
• Is the legal department prepared?

Assuming you decide to do something, there’s the decision set on how to get there:

• Build versus buy?
• Outsource versus in-house development?
• Cloud versus on-premises versus on-device?
• What front-end devices for the user interface?

Project Priorities and Goals

Why are you doing a generative AI project? When you’re trying to put out an AI project, whether to the general public or internally to your staff, there are two main competing priorities:

• Fear Of Missing Out (FOMO)
• Fear Of Fouling Up (FOFU)

Both are very valid concerns! Take too long and your competitors will be ahead of you. Even worse, there’ll be someone in a hoodie you’ve never heard of with a multi-billion dollar startup.

But there are also many examples of public failures and the consequent PR embarrassment from newly-released AI projects. Things like telling people to use glue on their pizza, or it’s fine to pick mushrooms without any concerns, or insensitively-written obituaries.

Goals. The goals of your project might be:

• Company reputation
• Market positioning
• Growth (of revenues, users, market share, etc.)
• Productivity
• Cost-reduction
• User empowerment (adding capabilities)
• Customer requests

And there may also be defensive goals:

• Not falling behind in tech
• Fighting the disruptors
• Staff morale

These are all laudable goals, but some of them are tough to achieve. Cost reduction and productivity improvement is a tactical goal, whereas protection against disruptors is a long-term strategic goal.

Choosing Your AI Project

What’s the project? Here are some examples of common projects for business usage of AI:

• Writer copilot for marketing or other departments.
• Coding copilots for software developers.
• Support chatbot for your website, that directly answers customer questions about your products.
• Q&A internal service for support staff to help answer questions and offer “scripts” to follow.
• In-house Q&A service to answer sales staff questions about your products (with more in-depth answers possible than a public chatbot)
• In-house HR chatbot to answer staff questions about HR policies and internal company matters.

The above are all quite large projects. One approach is to find value in automating much smaller areas for incremental improvements to workflow. Some examples in the IT department could include:

• Post-processing error messages and alerts by passing the error message through ChatGPT. This can make them more readable, and offer “expert advice” on how to fix them.
• Generating minor technical formats, such as JSON configuration files or rules, for whatever IT software you’re already using (e.g., for Prometheus, Graylog, etc.).
• Generating SQL queries from a text description, to query whatever databases are lying around.
• Writing small Linux shell scripts for whatever is needed.

Pro tip: you can choose more than one AI project!

A common type of first AI project is to get certain groups of staff trained up with AI tools to improve their productivity. These are the various “copilot” types of AI tools, and they can be used by various different company teams, even programmers. An important distinction here is that such copilot tools may not require the LLM to have any training with specific in-house data, which is another reason they can be launched faster.

What are Other Companies Doing?

What’s everyone else doing in AI? On the one hand, there’s a firehose of press articles about generative AI. On the other hand, many of the major players are startups that are still private, with opaque financials. Nevertheless, I’ve tried to accumulate some useful data points.

Follow the money. Who’s making money from generative AI? So far, some of the major winners where the information is available include:

• NVIDIA — H100 chips.
• OpenAI — over $3 billion in revenue.
• Microsoft — Azure and also various copilots.
• Accenture — over $2 billion in AI project bookings, versus $300m a year ago.
• Scale AI — approaching $1 billion in revenue.

OpenAI revenue. It is unclear whether OpenAI is making coin from individuals or businesses. However, they just made the consumer version free to use without a login (i.e., they just “ungated” access), and I doubt they’d do that if consumers were their main revenue. Some estimates are that the OpenAI API is the greatest source of revenue, which means business customers and other AI-first startups.

Scale AI business revenue. Scale AI, which is a startup that specializes in data cleanup and data labeling for AI, provided some insightful public information when they raised some more funding. Recently, they announced a massive increase in revenue, and also the change in that 90% of recurring revenue is now driven by corporate customers. Previously, their main customers were the big AI startups building foundation models.

General market status. My analysis of the above points and from reading many, many articles is that generally, the situation is this:

• Lots of companies buying or renting H100 chips (and now B100s)
• Lots of spending on consultancy services (unclear to what extent it’s advice or build-it-for-me projects).
• Lots of “buy” rather than “build” for internal staff productivity (e.g., writing copilot products, coding copilots, Microsoft Copilot, etc.)
• On-device AI for phones (Apple and Android) is just starting.
• AI PC on-device applications just starting (e.g., Microsoft and Copilot+ PCs).
• AI security products.

Business project status. In terms of business-specific AI development projects, my conclusions from this:

• Lots of businesses are doing AI projects (or at least “experimentation”).
• About 2 years into a 3-5 year deployment cycle of specialized business AI applications.
• Many projects are stuck in POC and pre-production (but this is changing).
• Employees aren’t waiting for their company to catch up, and are using consumer AI products in their work on a massive scale (and leaking lots of your proprietary data!).
• Building AI apps is a struggle because nobody knows what to do — too many complicated issues, and too many vendors.
• Safety issues and regulatory compliance are another pain point slowing adoption.
• Dirty data from inside the company is a major bottleneck for many business AI projects.
• Public-facing external AI applications are much less common than internal usage.
• Business-specific AI “platforms” based on internal competencies are rare.

This all sounds great, but there’s really only one big problem.

Confusion Reigns

Well, at least the consultants are doing well. Presumably, that’s because nobody else really knows what to do. And just between you and me, I’m not sure that the consultants know all the answers either.

In order to help you out, here’s my list of the areas of confusion in building an AI application:

• What to build
• What data to use
• What platform to use
• What model to use
• How to build it
• How to test it
• Whether it’s legal

Maybe it’s better just to stay in bed. But at least you’re not alone. Everyone’s confused!

The Easier Options

In order to take it easier on yourself, consider the simpler projects first. Here are some thoughts on easier ways to go into AI. Note that I didn’t say that they are the cheapest ways, or the most beneficial. All I said was “easier” and I definitely did not say “easy.”

Here are the easier types of projects to choose:

• Buy is easier than build.
• Internal usage is easier than customer-facing.
• Text is easier than images, which is easier than video.
• Human-in-the-loop is easier than unattended or automated (i.e., a real person reviewing AI’s outputs makes it easier to avoid pitfalls).
• Read before you write (i.e., choose projects that just answer questions, but don’t “change” anything).
• Web-based browser interfaces to cloud backends are easier than on-device AI.
• AI agents are hard and very new.
• Building your own AI platform is the worst.

To the last point, the hardest project is building your own “platform” that allows you to extend easily into multiple business-specific AI applications. Although it’s tough to do, this idea has the greatest long-term strategic benefit.

Internal Business AI Platforms

There are a lot of short-term wins in staff productivity and addressing common customer pain points. A lot of these are “buy” rather than “build” and the tendency will be to de-prioritize the more complex AI projects. However, these short-term projects are not the big strategic wins that realize the promise of AI. That can be summed up as: a business-specific AI platform.

What’s a business AI “platform”? Well, in the same ways that the internet is widely used internally by business staff and externally to communicate with customers, an AI platform will generalize that idea. The major capabilities include:

• Internal business processes (simplified, automated, extended)
• Customer communication (personalized, automated, integrated)
• Data analysis (deeper patterns, timeframe planning, niche segmentation)

Making AI into a core competency within your organization is a long-term strategic advantage. Every business is different, and business-specific requirements can go well beyond the generic capabilities of writing copilots, customer support chatbots, and other off-the-shelf AI systems.

In assessing your long-term strategy for AI, consider what makes your company unique. What are your strategic advantages against competitors? Then consider how generative AI could enhance and extend those capabilities, looking at both inside and outside the company.

AI Core Competency Levels

It’s very complicated at the moment, but there seems to be a natural progression occurring. The distinction can be mapped as:

• Tactical AI projects
• Strategic use of AI

Here are my thoughts on a hierarchy of AI core competency levels inside a business.

Tactical AI. For short-term AI projects, the levels are:

• Buy for internal staff — e.g., writer copilots, coding copilots, or other turnkey AI applications.
• Buy for external customers — e.g., a “custom” customer support chatbot based on RAG or fine-tuning from auto-ingesting your website.
• Build for internal staff — e.g., automate some of the more annoying internal repetitive business processes for improved staff productivity.
• Build for external customers — e.g., clean up internal proprietary data and use this for better fine-tuning of specific customer-facing apps.

The goals of tactical AI projects are likely to be specific and measurable. Examples include:

• Cost reduction
• Productivity improvement (e.g., words written, SLOC, etc.)
• Customer satisfaction metric improvement (e.g., average time to resolve customer questions).
• Conversion rates (e.g., personalized sales content in customer outreach).

Strategic AI. Well, the tactical AI projects have been going on for a while. If you really want to get ahead in the longer term, think strategy. The strategic competency levels are much more difficult:

• Internal business-specific AI platform — extend your capabilities to allow easy creation of many “mini-apps” for business process automation.
• External business-specific AI platform — control the touch points with your customers in a way that you can easily leverage and extend.

At the strategic level, the goals will be less quantitative, and longer-term:

• Repeatability — quickly bringing new capabilities online using the same process and underlying framework.
• Scalability — being able to scale onboarding quickly, whether it’s staff or customers.
• Specialization — use of your organization’s core competencies.

Running your own AI platform is the biggest long-term win. I’m not sure that you need to run your own H100s, and you might simply build out your own platform by customizing and extending an existing AI infrastructure platform. But I think it’s important to control the ability to add higher-level functionality in your core business areas. By owning the innovation level, you get to drive it forward with specialized capabilities that your competitors will struggle to match. It takes time and money, but this will be the long-term battleground in generative AI.

In order to avoid sending too much of your cash to the hyperscalers while building your AI platform, there are some ways to reduce costs. Early on, and for the development of many features, mock the AI part. A local AI with a tiny model that conforms to the OpenAI APIs will work, so the whole app can be developed, debugged, and its many parts tested without paying by the token or risking exposure of any IP. Once a criteria is met, switch over to the “real” AI and start doing the integration testing properly.

Weirdly, the opposite plan also works to some extent, too. The queries/prompts can be created and validated ahead of time using the full AI model (e.g., ChatGPT interactive), before development starts, even just to prove the value of the AI approach. This kind of “scoping” or “feasibility” work need not be performed by developers, so it could be given to marketing. These AI actions will cost a relatively small amount of money, and you can POC the AI features before you try to develop them!

References

1. Gene Rapoport, Sanjin Bicanic, Jue Wang, Richard Lichtenstein, Arjun Dutt, June 20, 2024, AI Survey: Four Themes Emerging: If 2023 was about experimentation, 2024 is all about results. Bain & Company, https://www.bain.com/insights/ai-survey-four-themes-emerging/ (Bain reports that use cases have been broadly successful in the use cases of sales, sales operations, software development, marketing, customer service, and customer onboarding, but less successful in HR, operations and legal. Interestingly, the main reason for AI project failures was that it couldn’t perform the necessary task.)
2. Timothy Mugayi, Sep 2024, LLM Practical Ideas to Build Your Next AI-Powered Application: Realistic Use Cases to Unleash the Power of AI in Your Next Project, https://levelup.gitconnected.com/llm-practical-ideas-to-build-your-next-ai-powered-application-9379feba6cbc
3. Irene Weber, 13 Jun 2024, Large Language Models as Software Components: A Taxonomy for LLM-Integrated Applications, https://arxiv.org/abs/2406.10300
4. Chuan Yan, Ruomai Ren, Mark Huasong Meng, Liuhuo Wan, Tian Yang Ooi, Guangdong Bai, 26 Aug 2024, Exploring ChatGPT App Ecosystem: Distribution, Deployment and Security, https://arxiv.org/abs/2408.14357
5. A16Z, April 2nd, 2024 (accessed), AI Getting Started, https://github.com/a16z-infra/ai-getting-started (Javascript wrapper kits for several commercial AI APIs.)
6. Joe McKendrick, March 28, 2024, This year’s top 8 use cases for AI, and what tech professionals need to support them, https://www.zdnet.com/article/this-years-top-8-use-cases-for-ai-and-what-tech-professionals-need-to-support-them/
7. Rafe Fletcher, July 11, 2024, Finding AI’s Killer Use Case, https://www.linkedin.com/pulse/finding-ais-killer-use-case-rafe-fletcher-7welc/
8. Stephanie Houde, Vera Liao, Jacquelyn Martino, Michael Muller, David Piorkowski, John Richards, Justin Weisz, Yunfeng Zhang, 2 Mar 2020, Business (mis)Use Cases of Generative AI, https://arxiv.org/abs/2003.07679
9. Olivia Moore, March 13, 2024, The Top 100 Gen AI Consumer Apps, https://a16z.com/100-gen-ai-apps/
10. MV Financial, Jul. 20, 2024, The Search For The Killer AI App, https://seekingalpha.com/article/4705253-search-for-killer-ai-app
11. David Linthicum, Feb 13, 2024, 3 killer apps for cloud-based generative AI, https://www.infoworld.com/article/2335997/3-killer-apps-for-cloud-based-generative-ai.html

• Online: Table of Contents • PDF: Free PDF book download • Buy: Generative AI Applications: Planning, Design and Implementation

The new Generative AI Applications book by Aussie AI co-founders: Deciding on your AI project Planning for success and safety Designs and LLM architectures Expediting development Implementation and deployment Get your copy from Amazon: Generative AI Applications