ML vs LLM: 5 Prompts That Break LLMs — and How ML Tool-Calling Fixes Them

TL;DR

• Your LLM can run Python now, so it can compute. It still can’t learn a model that generalizes, hold per-stream state, or score a live feed. Computation was never the hard part.
• ML models are precise at narrow numerical tasks and useless at deciding what to do with the result.
• A useful AI agent needs both. Below: a decision matrix and five prompts that break an LLM, each with the ML tool-call that fixes it.

Why teams keep reaching for the wrong tool

The ML-vs-LLM question comes up more than almost any other in applied AI work — and nine times out of ten the problem isn’t the data. It’s that a team wired an LLM to a job that needs ML, or built an ML-only dashboard that needed an LLM to make it worth reading.

It’s an easy trap. LLM demos are dazzling: describe a problem in plain English, get something coherent back. So the natural next step is to pipe sensor data into that same model and ask it to forecast tomorrow’s load, flag the failing pump, or rank machines by risk. It works beautifully — in the demo. Then production arrives with noise, gaps, and drift, and the confident answers quietly go wrong.

The opposite failure is just as common and quieter: precise ML scores sitting in a dashboard nobody opens. The interesting work is at the seam between the two — which is exactly where someone pushes back:

Wait — can’t LLMs just run Python now?

Fair challenge. Two years ago the answer to “why not just ask the LLM?” was “it can’t do the math.” That answer is now wrong — and if this article still leaned on it, a sharp engineer would close the tab. So let’s be precise about what changed and what didn’t.

In a chat window, yes. ChatGPT’s Advanced Data Analysis and Claude’s analysis tool both write and run Python in a sandbox. But that sandbox lives in the product wrapped around the model — not in the model itself.

Through the API, only if you wire it up. The API is the layer almost every IoT agent and platform is actually built on, and there code execution is opt-in. OpenAI’s Code Interpreter is a tool you pass in the tools array and back with a provisioned container (~$0.03 each); Anthropic’s code_execution tool on the Messages API is beta and off by default. Call the plain endpoint with a prompt and you get tokens, not execution.

Inside a production agent, it’s the wrong shape anyway. Even enabled, what you get is an ephemeral sandbox: spun up, hands back a result, torn down. It’s a calculator, not a place to train a model, hold a per-sensor baseline, or watch a live stream. And it runs blind — the code is written fresh each call, imports whatever happens to be in the sandbox image, and unless you build the plumbing to capture it, executes with no reviewed, reproducible trace. For a number that dispatches a technician or trips a safety alarm, “the LLM wrote some Python we never looked at” is disqualifying on its own.

So the honest rule isn’t “LLMs can’t compute.” It’s this: code execution lets an LLM compute; it does not let it learn a model that generalizes, hold state across hundreds of streams, or score a feed in real time. That’s the line — and it’s exactly the line the five prompts below cross.

The real dividing line: compute vs. learn

ML learns a model that generalizes; the LLM reasons about the result in language. Code execution lets the LLM compute — it does not let it learn.

ML for inference, LLM for reasoning.

ML is the right tool when the question is "what's the number?" — a forecast, an anomaly score, a remaining-useful-life estimate, a class label. LLM is the right tool when the question is "what should we do about the number?" — explain it, prioritize it, draft the message, decide the next step, call the right tool.

Three quick sanity checks:

• Can an LLM forecast a time series? Sometimes — badly. Not for production workloads where being wrong costs money.
• Can an ML model write a maintenance ticket? No. It returns a number. What the number means in context is a language problem.
• Do I need both? For any workflow with a human in the loop, yes.

The decision matrix

This is the ML vs LLM table we keep redrawing on whiteboards for IoT and industrial projects — when to use traditional machine learning, when to use an LLM, and when to use both. Bookmark it.

If your task is a row in the left column and you're reaching for the wrong tool in the middle column, this article is for you.

Five prompts that break LLMs

Every example follows the same shape: the prompt that breaks — why running Python doesn’t save it — ML’s job (with a code sketch) — the handoff the operator actually sees.

4.1 Anomaly detection — vibration on a pump motor (manufacturing)

The question: Is this pump about to fail?

Hand the raw series to a code-running LLM and it’ll compute means, maxima, and a tidy standard deviation. But “about to fail” isn’t a calculation — it’s a deviation from a learned model of this pump’s normal, and the LLM has never seen this pump.

ML's job. Train a machine-learning model — an Isolation Forest — on 90 days of "normal" vibration RMS and bearing temperature. Score new readings in real time.

from sklearn.ensemble import IsolationForest
import numpy as np

# X_normal: shape (n_samples, 2) — [vibration_rms, bearing_temp]
clf = IsolationForest(contamination=0.01, random_state=42)
clf.fit(X_normal)

# Negative scores = more anomalous. Threshold at -0.1 works for most industrial sensors.
scores = clf.score_samples(X_today)
anomalies = X_today[scores < -0.1]

The model returns a score per minute. It has no opinion on what to do with it.

LLM's job. When a score crosses the threshold, the LLM pulls the work-order history for that asset, checks the last service date, queries parts inventory, and composes a message for the maintenance lead.

The handoff the operator sees:

"Pump P-103 crossed its anomaly threshold at 02:14. Last bearing service was 7 months ago — overdue per the 6-month maintenance plan. No spare bearing kit in on-site inventory. Recommend dispatching a tech this shift before the morning run."

The ML model produced a number. The LLM turned it into a decision. This pattern is the core of what makes smart manufacturing AI deployments actually useful on the shop floor.

4.2 Forecasting — energy demand in a building (HVAC / facilities)

The prompt: “How much energy will this building draw tomorrow?” A code-running LLM will fit a quick regression on whatever you paste and hand back a confident, specific number — with no memory of your building’s baseline. It’s pattern-matching what consumption sounds like, not forecasting it. This isn’t speculation — a 2024 NeurIPS paper found that stripping the LLM out of popular LLM-based forecasting methods didn’t hurt accuracy and often improved it.

ML’s job. Lag features (hour-of-day, day-of-week, lag-1d, lag-7d, temperature forecast) into an XGBoost regressor — or Prophet if a non-ML engineer has to maintain it.

import xgboost as xgb
from sklearn.model_selection import train_test_split

# features: hour, dow, lag_1d, lag_7d, temp_forecast
X_train, X_test, y_train, y_test = train_test_split(features, target, test_size=0.2)

model = xgb.XGBRegressor(n_estimators=200, learning_rate=0.05)
model.fit(X_train, y_train)

tomorrow_forecast = model.predict(tomorrow_features)

The handoff. The LLM turns the number into an instruction — “Tomorrow is 14% above a typical Wednesday, likely the forecast heat wave. Pre-cool the building 3–5am to shift load off-peak and dodge the demand charge.”

4.3 Predictive maintenance — remaining useful life (logistics / mobility)

The prompt: “Which trucks need their alternator replaced in the next 30 days?” That’s a survival model fit to historical failures plus current degradation signals — voltage drift, restart frequency, heat exposure — ranking assets it has never seen. An LLM can’t extrapolate that.

from lifelines import CoxPHFitter
import pandas as pd

# df: one row per asset, columns = degradation signals + 'duration' + 'event' (1 = failed)
cph = CoxPHFitter()
cph.fit(df, duration_col='duration', event_col='event')

risk_scores = cph.predict_partial_hazard(current_fleet_signals)
ranked_fleet = current_fleet_signals.assign(risk=risk_scores).sort_values('risk', ascending=False)

The handoff. The LLM ranks the model’s output against this week’s business context — mission-critical routes, which depots stock the part, who’s on shift — and drafts the top-five work orders. That’s the core of IoT predictive maintenance: the ranked list alone gets ignored; the ranking plus reasoning gets actioned.

4.4 Classification — water-quality alarms (utilities / environmental)

The prompt: “Is this pH / turbidity / conductivity combination a real alarm or sensor noise?” Most water infrastructure monitoring still uses fixed thresholds that over-fire by three to five times; a classifier trained on ~10k labelled readings more than halves the false positives. Asking an LLM to hand-label all 50,000 daily readings instead isn’t consistent run to run and costs far more than a batch that scores in milliseconds.

from xgboost import XGBClassifier

# X: [pH, turbidity, conductivity, hour_of_day, sensor_age_days]
# y: 1 = real alarm, 0 = noise
clf = XGBClassifier(n_estimators=100, use_label_encoder=False, eval_metric='logloss')
clf.fit(X_train, y_train)

p_alarm = clf.predict_proba(X_today)[:, 1]

The handoff. When P(alarm) > 0.7, the LLM pulls upstream context — recent rainfall, nearby maintenance, neighboring sensors — and decides: dispatch a field sample, open a ticket, or auto-close as noise.

4.5 Free-text → structured — technician shift notes (the one where LLM wins outright)

The question: Across thousands of unstructured technician notes written over two years, what's actually breaking out there?

Why ML loses here. You'd need a labelled training set you don't have, and the categories shift over time as equipment ages and new failure modes appear. A supervised classifier trained six months ago won't catch the new failure mode that started appearing last quarter.

LLM's job. Zero-shot or few-shot classification into a stable category list — "electrical fault", "mechanical wear", "comms loss", "user error", "unclear" — plus extraction of asset ID, severity, and any parts mentioned.

import anthropic

client = anthropic.Anthropic()

def parse_note(note_text: str) -> dict:
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=256,
        messages=[{
            "role": "user",
            "content": f"""Classify this technician note. Return JSON only.

Note: {note_text}

Schema: {{
  "category": "electrical_fault|mechanical_wear|comms_loss|user_error|unclear",
  "severity": "low|medium|high",
  "asset_id": "string or null",
  "parts_mentioned": ["list of strings"]
}}"""
        }]
    )
    import json
    return json.loads(response.content[0].text)

No training data. No labelling sprint. Ship it this week.

The handoff runs in reverse here. The LLM produces the structured fields → plain SQL (or an ML model, if you have enough rows) aggregates them into a Pareto chart of fault categories by site, by asset class, by quarter.

Honesty check: if you already have 100k labelled notes and a stable category taxonomy, fine-tune a small model — it'll be faster and cheaper per call. If you don't, the LLM is unambiguously the right starting point.

Two more that break — for a different reason

“Which of my 400 sensors are behaving abnormally right now?” The killer here isn’t the math — it’s 400 and right now. There’s nowhere in an LLM to keep a live, per-sensor baseline for 400 streams and score each new reading against it. That’s persistent state, and the LLM has none.

“Alert me the moment this reading deviates from normal.” An LLM is request—response. It can’t sit on an MQTT stream and score every packet the millisecond a threshold trips. This one isn’t hard, it’s architecturally impossible — no amount of code execution changes it.

The architecture pattern

Put it all together and the diagram is the same regardless of vertical:

The LLM is the orchestrator. The ML models are tools. So is the database. So is Slack. Treat them all the same way — as callable tools the LLM can invoke — and the architecture stays clean regardless of how many models or data sources you add. This is the same architecture at the heart of modern cloud-based SCADA and Industry 4.0 systems — the difference is that the reasoning layer is now an LLM instead of a rules engine.

One thing this diagram makes obvious: the LLM never touches raw data. It works on aggregates, scores, and summaries that the ML and data tools produce. That's by design. An LLM reasoning over a 50-row summary is fast, cheap, and reliable. An LLM pointed at 50,000 raw rows — even one running its own Python — is slow, expensive, and stateless: it recomputes from scratch every call instead of querying a model that already learned the pattern.

Anti-patterns

1. Asking the LLM to write SQL on large tables. The LLM can write SQL fine. The problem is running a complex query against 25k rows and returning all of it to the context window — slow, expensive, error-prone. Aggregate first with a “prepared view” tool that collapses the raw data to the 50 rows that matter, then let the LLM reason over the summary.

2. Baking business rules into the ML model. If the model is trained to flag a reading as an alarm given your current SLA thresholds, every threshold change sends you back to retraining. Keep business rules in the LLM prompt layer, where editing is a one-line change, not a retraining cycle.

3. Treating a demo as a proof of concept. The demo worked on 10 clean points in a notebook. Production has 10,000 points, missing timestamps, duplicate readings, and sensors that drop for three days and come back with stale values. A trained pipeline handles that; a prompt — even one with a code sandbox — does not, because it never learned what your clean signal looks like.

FAQ

Can’t LLMs just run Python now — so why do I need ML?
Code execution fixes computation, not learning. And it’s usually not even in play: through the API it’s an opt-in tool most agent platforms never wire up. Even where it is, it’s an ephemeral sandbox — it can’t train a model that generalizes, hold per-stream state, or score in real time, and its fresh, un-pinned, non-deterministic code isn’t auditable the way a production number must be. An LLM can calculate over data you hand it; it can’t be the model.

When should an AI agent use ML instead of an LLM?
Whenever the task is "produce a number." Forecasting, anomaly scoring, classification, and remaining-useful-life estimation are all ML tasks. An LLM — even with a code sandbox — can compute a one-off value, but it can’t be the model: it can’t learn a pattern that generalizes, hold state, or run in real time.

Can an LLM do time-series forecasting?
In a limited sense, yes — for very short horizons on simple patterns, an LLM with a tool-call to a statistical model can appear to forecast. But it is not doing the forecasting; it is calling the tool. Asking a raw LLM to extrapolate a numerical sequence from context will produce confident-sounding, statistically wrong results. Don't do it in production.

Can an LLM do anomaly detection?
Not reliably. Anomaly detection requires a learned model of "normal" behavior built from historical data — a distribution, a baseline, a threshold calibrated to your specific equipment. An LLM has no such model of your specific equipment. It can reason about an anomaly score once an ML model has computed one.

What's the difference between an AI agent and a machine learning model?
An ML model is a function: it takes structured input and returns a number or a label. An AI agent is an orchestrator: it takes a goal, decides which tools to call (including ML models, databases, APIs, messaging services), interprets the results, and takes action. The agent without ML tools is a chatbot. The ML model without an agent is a dashboard nobody reads.

Do I need to retrain my ML model when business rules change?
No — and this is one of the most underappreciated benefits of the two-layer architecture. Keep business rules (thresholds, SLAs, escalation logic, customer-specific policies) in the LLM prompt layer. They're plain text; changing them takes seconds. Reserve retraining for when the underlying signal distribution changes — new equipment, new failure modes, new operating conditions.

Is fine-tuning an LLM a substitute for an ML model on numerical tasks?
Almost never. Fine-tuning teaches the LLM to produce text that looks like the right numerical output. It does not teach it to compute. For production workloads where numerical precision matters — forecasting, anomaly scoring, classification — a well-tuned scikit-learn model on clean features will outperform a fine-tuned LLM, run faster, cost less, and be easier to audit.

One more thing

This pattern — AI in IoT done right — shows up constantly in what we build at Ubidots. Our AI Agents capability is the LLM-orchestrator side of this picture, and UbiFunctions is where you drop your own Python — scikit-learn, XGBoost, pvlib, your own trained model — as a callable tool. Same architecture as the diagram above; we just give you the runtime and the sensor data already wired in — the persistent, auditable home an ephemeral LLM sandbox will never be.