Aussie AI

Chapter 2. Generative AI Market Overview

  • Book Excerpt from "Generative AI Applications: Planning, Design and Implementation"
  • by David Spuler

Chapter 2. Generative AI Market Overview

The State of AI

There’s so much going on in the AI industry that these words are out-of-date the second that I type them. Nevertheless, here are a few general thoughts on where we are:

AI is popular. Even two years after the launch of ChatGPT, generative AI is still the talk of the town (and the boardroom!). With new capabilities launching often, there’s much to keep everyone entertained.

AI is amazing. Already, it seems like “just” writing fluent text is old hat. But I’m still astounded by the capabilities of the latest AI apps with vibrant realistic images and alluring video clips. There are so many advances happening quickly in speech, vision, animation, and video. The whole industry is evolving rapidly at such speed that I want an AI copilot to help me keep up with all the news.

AI is improving rapidly. The leaderboards for AI model intelligence continue to post higher scores for new models in terms of base intelligence and complicated reasoning. There’s also better handling of negative issues such as safety, toxicity, and bias.

AI is expanding. ChatGPT launched as text only, but now we have image intelligence widely available, and there’s starting to be video, too. Multimodal models such as GPT-4o and Google Gemini can both output and understand images and text, together or separately.

AI is expensive. Remember the joke about how “boat” stands for “Bring Out Another Thousand”? That’s nothing compared to AI. A single GPU costs more than your boat and a typical motherboard has eight of them. And the big tech companies have been buying these by the thousands. What should LLM stand for? “Lavish Leviathan Mammoth”? That was mine. “Ludicrously Large Mango” was Bing Chat with GPT-4’s AI suggestion. Neither are great, which is comforting because it means there’s still some work to be done.

AI is not new. The AI-related workload hosting market is many years old. Just because generative AI has blasted into consumer consciousness, and into boardroom discussions as a result, doesn’t mean that AI is new. The cloud hosting companies like Amazon AWS, Microsoft Azure, and Google GCP, have been doing AI workloads for many customers, for many years. Instead of using GPUs for generate AI, they’ve been running workloads in other AI areas like Machine Learning (ML), machine vision (e.g., Tesla autonomous cars), product suggestion feeds, predictive modeling, auto-completion of search queries, and so on. There were already billions of dollars invested in AI long before ChatGPT set the web on fire.

AI Phones. AI is going to be on your phone, and it’s going to be a big driver of new phone purchases. Google has the Android on-device SDK to build phone apps, and Apple has launched the “Apple Intelligence” models for on-device inference on iPhone.

AI PCs. AI models and applications are set to make PCs hot again in the near-term. The next generation of laptops and desktops will run AI models natively, and there will also be hybrid architectures with AI workloads offloaded into the cloud. Microsoft has launched its Copilot+ PCs, and Apple Intelligence has been announced for MacOS.

Green AI. The widespread use of AI makes it a significant contributor to energy consumption, and there is much research on the environmental impact from AI computing. On the positive side, this means that all of the research towards AI improvements is helpful for green AI, since it will also reduce its carbon footprint and environmental impacts. All of those kernel optimizations to speed up the AI engine are also making things greener overall.

AI Market Trends

Here are some future-looking thoughts about what the market for AI may look like. It seems likely that programmers will be required for a little while longer.

It’s a marathon, not a sprint. Consumers may continue to adopt generative AI quickly, but that’s not the most likely case for businesses. Whereas generative AI is a hot topic in boardrooms, most business are still trying to find their feet in the area, with only exploratory projects launching. Small businesses and professionals (e.g., doctor’s offices) will take years to adopt generative AI, and larger enterprises will take even longer. There will be some early projects, sure, but the bulk of the B2B AI market will evolve more slowly. Projections for the B2B side of AI are over many years, even decades, with high CAGR. We’ve already seen this in the business adoption of cloud architectures, which is still ongoing, despite having been running since the early 2000’s. The B2B AI market is likely to sustain very strong growth through 2030 and probably even into the 2040s and beyond.

B2B market opportunity trumps B2C. The massive ramp-up of consumer engagement with ChatGPT has made the consumer side seem hot. However, it’s more likely to be the business side that makes more money (as usual). Predictions of the billions, maybe trillions, of dollars of benefit to economies through full AI integration into businesses, dwarf the predictions for consumer opportunities.

Training is the big B2B market? Early wisdom was that the high cost of training and fine-tuning would far exceed inference costs. This contention is somewhat in dispute, with some pundits saying that the sheer number of users will push inference ahead of training. Another factor is the trend toward using someone else’s pre-trained LLM, whether it’s GPT via the OpenAI API or the open source Llama models. Hence, there’s definitely more inference than training in B2C projects, and it may also be taking over on the B2B side.

Fine-Tuning vs RAG. Most business AI projects will involve enhancing the model using proprietary data that the business owns. For example, a support chatbot has to learn information on the company’s products, or an internal HR chatbot needs to use internal policy documents. There are two main ways to do this: fine-tuning or Retrieval-Augmented Generation (RAG). Current training and fine-tuning methods take a long time, need a lot of GPUs, and cost a great deal. However, RAG is becoming widely used to avoid the cost of fine-tuning.

Inference vs Training in the B2C market. Even the B2C generative AI bots need continuous training and fine-tuning, to keep up with current events, so there will also be significant training cost (or RAG costs) in the B2C market. However, with millions of users for B2C apps, the cost of inference should overshadow training costs in the long run.

Bull Case for AI

The bull case is, well, go and look at the NVIDIA stock price. The estimates of revenue available for generative AI are literally over a trillion dollars a year, and we’ve only spent maybe fifty billion on this infrastructure, so there’s a lot more to win.

Although there’s plenty of hype, there are also those who suggest it’s overhyped. The bear case is basically summarized as follows: yes, congratulations, you’ve spent over fifty billion dollars, and only got a couple back in return. It’s mega-overhyped, as anyone who follows Gartner should know, and the only company laughing on the way to the bank is NVIDIA. There’s no killer app for users, except, I mean, well, ChatGPT was only a little bit popular, so nobody’s going to care about AI soon.

The bull case is basically that there are two massive buckets of money up for grabs, kind of like in Hunger Games:

  • Business apps
  • Consumer apps

And, yes, the basic premise of this idea is that neither of these “app” areas have taken off yet, and are still in their infancy. Businesses are only now starting to figure out what they can use generative AI for, whether internally for staff productivity, or externally for their customers. Consumers are using generative AI, mainly for ChatGPT and AI companions, but there are still many new areas that will grow.

And then there’s the tech. Come on, be honest, it’s pretty amazing. Things like:

  • Understanding (of documents and your commands)
  • Writing (creatively or professionally)
  • Coding
  • Audio
  • Video creation
  • Animation
  • Music creation
  • Editing text or photos or videos or whatever...

The AI industry has some underpinning trends that affect all AI projects:

  • Price of AI inference is dropping fast.
  • Multi-AI ensemble architectures are smarter.
  • Multi-step reasoning algorithms (i.e., multiple LLM queries per user query).
  • Follow-up question asking (like a real personal assistant would).

And then it’s going to get even more techie:

  • Voice interfaces — your phone stays in your pocket.
  • Video understanding — complicated but doable.
  • Active autonomous agents — that “do” stuff for you, not just answering questions.
  • Context detection — what’s on your screen now that you’re looking at?
  • User presence detection — are you looking at your screen right now?
  • Background context analysis — are you in the kitchen?
  • Geo-location correlation — are you en route to the airport now?

And there are some massive new form factors and use cases that may broaden the applicability of AI applications (or maybe even displace the prevalence of the smartphone):

  • AI gadgets
  • Autonomous transportation
  • Robotics
  • Industrial automation

And then there are some other non-AI advances that are going to add to this imbroglio:

  • 5G, 6G, 7G — one per year, maybe.
  • NPUs on phones — offline on-device native AI (faster, private, always ready).
  • Next-gen silicon — AI-specific chips.
  • Memory bandwidth optimizations — tighter GPU/CPU/memory architectures.
  • Network optimizations — faster networking going beyond RDMA.
  • Quantum computing — half-dead cats and all that jazz.

Users are going to love this stuff! Imagine an intelligent personal assistant in your pocket, which you can speak to, and it answers back into your ear-pods or whatever they’re going to be called. I mean, actually intelligent, not the clunky dumb version that you’re used to, that’s so 2020s.

Bear Case for AI

Although the volume is less, there’s a growing debate about whether generative AI is overrunning its fundamentals. Have we gotten a little ahead of ourselves?

Summarizing the main points of the bear case for generative AI, we get:

  • Hype? — Just a little bit. It’s a bubble (so goes the view).
  • No revenue — nowhere near as high as investment.
  • User’s declining interest
  • No killer app (as yet).
  • Too expensive — for everyone except NVIDIA.
  • Oversupply — AI infrastructure will become a glut.
  • Bad for the environment — ten-fold increase in data center electricity usage per query.
  • NVIDIA’s valuation — it’s too high to justify and NVIDIA would need to “grow into it” by earning even more revenue.
  • Limitations — generative AI has many limitations, and just isn’t that useful in reality.
  • Languishing business projects — stuck in “proof-of-concept” status.
  • Bad stuff — many examples of risks, false information, hallucinations, dangers, bias, inaccuracy, and outright stupidity.
  • Unimportant — disrupted industries are not high-value; writers and artists never got paid well, anyway.
  • Probabilistic — non-deterministic algorithms are not desirable for life-critical decisions.
  • Legal issues — e.g., bias, copyright, plagiarism, and many more billable hours.

Are you convinced? Do you agree that AI has run out of steam? Or maybe not just yet...

Bull rebuttals. Let us examine some of the bear case in more detail. Some of these points are valid, or have been valid in the past year, but things are changing. Hence, some comments:

  • Business projects are emerging from POC into production at an increasing rate as businesses (and technologists) are figuring out how to resolve the issues.
  • Demand for GPUs and other infrastructure will grow with increased adoption by both consumers and business, and also from newer GPU-hungry multi-AI architectures combined with multi-step reasoning algorithms.
  • Revenue growth: the revenues of OpenAI have doubled in less than six months and there are several other companies minting coin in AI companion bots, writer copilots, coding copilots, and other areas.
  • Limitations of generative AI continue to fall by the wayside as an army of researchers tackle them.
  • Safety issues and other concerns are reducing due to better training data and improved safety evaluation algorithms.
  • Probabilistic algorithms are problematic in some cases (e.g., latency-criticality in a pacemaker implant), but in terms of smartness, consider that the bio-brain of your pilot or doctor uses basically the same type of algorithm, except it’s in carbon not silicon.

Bear rebuttals. The above is somewhat one-sided, so let’s have a final look at some of the other negative points again:

  • GPU demand faces headwinds from research on (a) faster GPUs, CPUs, and NPUs; (b) optimizing software AI algorithms, which is an area I really like and think its potential is undervalued, and (c) on-device inference on AI phones and AI PCs that don’t use big GPUs.
  • Environmental impact of AI remains a real concern, unless said research really gets those electricity costs down.
  • Safety issues and reasoning limitations of generative AI are somewhat inherent to how it works, and hence, it’ll be hard going to resolve them all in a systematic way.
  • Many things are still unproven and mostly experimental, such as agent architectures and voice interfaces.

If nothing else, it’s going to be a fun ride!

Hype Versus Adoption

As everyone knows, new technologies go through the Gartner hype cycle. This involves initial hype, then a pullback, followed by a slower but more powerful increase in adoption that offers real benefits.

That seems likely to be the case for generative AI, but I wouldn’t want to try to guess the timing of any decline. There are some major trends that underpin growth at the moment and will further fuel the need for more generative AI technology work:

  • Consumer adoption is increasing (and for employees, too!)
  • New advances are dropping about monthly — e.g., text-to-video, avatars.
  • Voice interfaces — making it easier on the fingers and more natural to use.
  • Multi-AI capabilities — two AI engines are smarter than one, but need twice the juice.
  • Vertical-specific products — industry-specialized models are better.
  • On-device AI apps for phones and PCs — still a lot to build in this space.
  • Increasing intelligence — every extra percentage point in model capability is coming at a higher marginal cost for both training and inference.

And the other thing that would spur even more usage: a killer app.

Killer Apps

One premise of the AI bear case is that there’s “no killer app” (yet). But this seems a little disingenuous given the massive adoption of generative AI by consumers, especially with ChatGPT being the fastest growing app in the history of history. Some of the current batch of “killer apps” in the consumer space include:

  • ChatGPT — used for writing drafts, asking questions, brainstorming, etc.
  • AI companion bots — Character.AI has insane traffic levels.
  • Image generation — e.g., for book covers!
  • AI search — much smarter answers to user queries on Google and Bing.

And there’s some more potential killer apps on the horizon:

  • Video auto-generation — amazing, but currently limited in terms of clip lengths.
  • Voice interaction with smart assistants — a smarter Siri with voice interface is the potential killer app on the iPhone with Apple Intelligence.
  • AI gadgets — these haven’t hit mainstream, but offer potential disruption of the smartphone’s current dominance.
  • Agents — not just answering questions, but doing things for you (probably with your pre-approval, at least to start with).

Consumer Super App? Will there be a super app, that rises above all the rest? Well, there already is one, and it totally dominates smartphone usage in China. It’s called WeChat, and there’s hardly any smartphones in China that aren’t running it right now, as you read this. Yet, it’s almost unknown in the West. WeChat runs on top of iPhone and Android platforms, and has its own layer of “mini-apps” and payment support.

It’s never caught on in the USA or Europe, and I don’t know why. But delivering on a WeChat clone for the USA is still a wishful dream of many tech companies, and the closest thing we have today is Roblox! Maybe generative AI is the chance for them to do so. If there’s a voice agent in your pocket that knows everything and can do anything for you, that might be it.

Enterprise Killer App? I think it’s fair to say that there’s not yet a clear killer app in the business use of AI. Well, I mean generative AI in particular, because predictive ML has become entrenched in various areas, such as recommendation engines for product ads and content feeds. There have been some clear tactical wins for generative AI in certain business-related use cases:

  • Customer support chatbots
  • Writer copilots
  • Coding copilots
  • Data transformation assistants

However, nothing massive has really changed in the core framework of how business operates. Some of the potential candidate areas for enterprise-wide advanced killer apps would seem to be:

  • ERP systems, but smarter.
  • Workflow automation
  • No code AI platforms
  • Voice interfaces

A lot of vendors claim a lot of things in these areas, but I’m sceptical that anyone has really nailed it. There are fundamental problems in business with the sprawl of disparate applications and data sources across business divisions and geographies.

Will there be an enterprise killer app? Huge wins with generative AI might get done by the vendors, with their development cost effectively spread across multiple businesses, but they could also be achieved within a single business leader, by building their own internal AI platform.

Remember, AI tech is new and still in its teething phase, so we can expect lots of crying. But it will grow and we don’t really know yet into what. This is your chance to do something amazing and be the best! Happy inferencing!

References

  1. Nicholas Grous, Andrew Kim, June 04, 2024, Generative AI: A New Consumer Operating System, https://www.ark-invest.com/articles/analyst-research/generative-ai-a-new-consumer-operating-system
  2. Daniel Sack, Lisa Krayer, Emma Wiles, Mohamed Abbadi, Urvi Awasthi, Ryan Kennedy, Cristián Arnolds, and François Candelon, September 05, 2024, GenAI Doesn’t Just Increase Productivity. It Expands Capabilities, https://www.bcg.com/publications/2024/gen-ai-increases-productivity-and-expands-capabilities
  3. Sean Michael Kerner, October 28, 2024, Enterprise AI moves from ‘experiment’ to ‘essential,’ spending jumps 130%, https://venturebeat.com/ai/enterprise-ai-adoption-surges-as-organizations-shift-from-experimentation-to-implementation/
  4. Taryn Plumb, October 28, 2024 , Gartner predicts AI agents will transform work, but disillusionment is growing, https://venturebeat.com/ai/gartner-predicts-ai-agents-will-transform-work-but-disillusionment-is-growing/
  5. Eugene Cheah, Oct 11, 2024, $2 H100s: How the GPU Rental Bubble Burst. H100s used to be $8/hr if you could get them. Now there’s 7 different places sometimes selling them under $2. What happened? https://www.latent.space/p/gpu-bubble
  6. Ashu Garg, Oct 25, 2024, Why OpenAI’s $157B valuation misreads AI’s future, https://foundationcapital.com/why-openais-157b-valuation-misreads-ais-future/ (Bullish on the “application layer” saying “The top of the stack is where I see the most promise. ...the most valuable companies of the AI era don’t exist yet.”... “The cloud era created over 20 application companies with $1B+ revenue. In AI, we believe this number could exceed 100.”)
  7. Kate Knibbs, Oct 28, 2024, AI Slop Is Flooding Medium, https://www.wired.com/story/ai-generated-medium-posts-content-moderation/
  8. Taryn Plumb, October 28, 2024 , Gartner predicts AI agents will transform work, but disillusionment is growing, https://venturebeat.com/ai/gartner-predicts-ai-agents-will-transform-work-but-disillusionment-is-growing/
  9. Will Lockett Nov 2024, Apple Calls BS On The AI Revolution, They aren’t late to the AI game; they are just the only sceptical big tech company. https://medium.com/predict/apple-calls-bullshit-on-the-ai-revolution-ae38fdf83392
  10. Grant Gross, 18 Jun 2024, Generative AI’s killer enterprise app just might be ERP, The Information, https://www.cio.com/article/2149673/generative-ais-killer-enterprise-app-just-might-be-erp.html
  11. CNBC, June 16, 2024, Apple’s AI killer is... the iPhone, https://www.cnbc.com/video/2024/06/14/apples-ai-killer-is-the-iphone.html
  12. James O'Donnell, May 1, 2024, Sam Altman says helpful agents are poised to become AI’s killer function, https://www.technologyreview.com/2024/05/01/1091979/sam-altman-says-helpful-agents-are-poised-to-become-ais-killer-function/
  13. Rafe Fletcher, July 11, 2024, Finding AI’s Killer Use Case, https://www.linkedin.com/pulse/finding-ais-killer-use-case-rafe-fletcher-7welc/
  14. Alistair Barr, May 16, 2024·, What’s the killer AI app for consumers? Google finally has a contender. https://www.yahoo.com/tech/whats-killer-ai-app-consumers-175056899.html
  15. MV Financial, Jul. 20, 2024, The Search For The Killer AI App, https://seekingalpha.com/article/4705253-search-for-killer-ai-app
  16. David Linthicum, Feb 13, 2024, 3 killer apps for cloud-based generative AI, https://www.infoworld.com/article/2335997/3-killer-apps-for-cloud-based-generative-ai.html

 

• Online: Table of Contents

• PDF: Free PDF book download

• Buy: Generative AI Applications: Planning, Design and Implementation

Generative AI in C++ The new Generative AI Applications book by Aussie AI co-founders:
  • Deciding on your AI project
  • Planning for success and safety
  • Designs and LLM architectures
  • Expediting development
  • Implementation and deployment

Get your copy from Amazon: Generative AI Applications