Is AI Engineering Just API Plumbing? (Or: Where Does the Real CS Live?)
A personal journey through confusion, abstraction layers, and trying to figure out where a software engineer actually belongs in the age of LLMs.
To be completely honest, I spent months feeling skeptical about the AI hype. Aren't we all just calling external APIs and gluing JSON strings together? Here is how peeling back the abstraction layers changed my perspective.
The Core Dilemma: Is This Just API Plumbing?
To be completely honest, I spent the last few months feeling pretty confused about where I fit in this whole AI shift.
Every morning on my feed, everyone is talking about "AI Engineers," autonomous agent frameworks, and multi-step reasoning loops. But when I actually looked at the code inside a lot of these hyped repositories, I couldn't stop asking myself a skeptical question:
“Isn't all of this still just calling OpenAI or Anthropic endpoints under the hood? Aren't we just stringing together prompt templates, parsing JSON strings, and pretending it's new systems engineering? Where does the real AI live?”
For a long time, my knee-jerk instinct was: if you want true AI, you have to go down to the metal. You have to be the person working on the raw physics and math of code—implementing Transformer attention blocks from scratch in PyTorch, writing custom CUDA / Triton kernels for matrix multiplication, optimizing GPU memory consumption, and running post-training alignment through RLHF, DPO, or GRPO.
To me, that felt like where the true AI lived. Anything higher up the stack felt like superficial wrapper scripting around someone else's model—writing 500 lines of glue code just to parse a 5-word response, or spending half a afternoon engineering a prompt that politely begs the LLM not to make up fake SQL tables.
Challenging My Own Assumption: LLMs as Cognitive CPUs
Then I had an architectural discussion with a senior engineer that made me stop and re-examine my entire mental model.
He asked me a simple question about systems design:
“When you build a distributed database, do you care more about how silicon gates inside the CPU flip bits, or do you care about consensus protocols, write-ahead logs, page caches, and memory barriers built on top of it?”
That hit me.
If we treat a Large Language Model not as a magic text generator, but as a non-deterministic, probabilistic "Cognitive CPU," the entire equation changes.
A raw physical CPU needs a deterministic operating system around it to manage memory, schedule tasks, isolate processes, and handle peripheral I/O. In the exact same way, a probabilistic LLM requires a deterministic software runtime around it to guarantee correctness.
Calling an API isn't the hard part—anyone can pip install an SDK and burn through $50 in API credits in five minutes. The hard part is building software that produces predictable, type-safe telemetry out of an inherently stochastic model that might randomly decide to return JSON wrapped in markdown code blocks.
Low-Level Model Mechanics
Optimizing token log-probabilities, loss curves, gradient norms, attention heads, and GPU memory bandwidth.
# PyTorch Attention Kernel
attn_weights = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(d_k)
attn_probs = F.softmax(attn_weights, dim=-1)
output = torch.matmul(attn_probs, v)Systems-Level Orchestration
Structuring schemas, state transitions, tool execution registries, fallbacks, and static type bounds.
# Pydantic AI Type-Safe Agent @agent.tool async def query_db(ctx: RunContext, sql: SQLQuery) -> QueryResult: validated = ctx.deps.verifier.check(sql) return await ctx.deps.db.execute(validated)
Frameworks like Pydantic AI, LangGraph, DSPy, and LlamaIndex aren't just convenient wrappers. They are static typing systems, stateful graph controllers, and compiled signature runtimes engineered specifically to bring deterministic guarantees to non-deterministic model outputs.
Peeling Back the Abstraction Layers
When you compare traditional web software with real agentic AI systems from first principles, three distinct mechanisms stand out:
Agents vs. Traditional Web Code
Traditional web applications execute static decision trees: an incoming HTTP request hits an endpoint, runs nested if/else checks, and calls hard-coded API chains.
Agentic architectures shift decision-making into semantic space. Instead of static branches, agents ingest context, evaluate intent, autonomously select tools from a registry, assess their own intermediate execution results, and attempt self-correction when an API throws an error. That isn't simple scripting—it's dynamic control flow.
Agents as Intelligent Controllers Over Databases
There's a popular myth that AI agents will somehow make relational SQL databases obsolete. But why would they? You don't use an LLM to replace fast B-tree index lookups or ACID-compliant joins.
We use agents as an intelligent controller positioned above relational and vector databases to handle human chaos. Human intent is messy and ambiguous; relational databases demand crisp SQL. The agent acts as a cognitive buffer—interpreting messy Text-to-SQL intent, resolving entity ambiguities, enriching dirty text, and reasoning through business edge cases before committing state to storage.
Lessons from the Ho Chi Minh City AI Challenge (HCMC AIC)
I'm seeing this firsthand right now—as of writing this post, I'm actively preparing and competing in the Ho Chi Minh City AI Challenge (HCMC AIC).
In these high-pressure settings, winning is never about training a 70-billion parameter model from scratch in a limited time frame. Victory comes down to system orchestration: multi-agent query expansion pipelines, indexing visual vectors across temporal keyframes, routing queries dynamically based on modality confidence, and executing sub-second hybrid searches across vector indices. System design beats brute-force training every single time.
My Strategic Path Forward: The Hybrid Engineer
So where does this leave me?
Through this reflection, I realized I don't have to choose between being "the low-level model trainer" or "the high-level systems architect." They aren't opposing paths—they are complementary layers of the same stack.
That's why I've been systematically grounding myself in traditional AI fundamentals—working through the mathematics, linear algebra, and classical deep learning architectures in the AI Viet Nam (AIO) program—while keeping my primary identity and focus as a production software engineer.
The strongest agent developers are those who deeply understand token mechanics, attention saturation, and quantization degradation under the hood. And the best model engineers are those who understand how downstream production systems consume their weights in real-world infrastructure.
MY_PATH_FORWARD // THE_HYBRID_ENGINEER
“My path forward to bridge both worlds is mastering deep learning fundamentals under the hood, while using my backend engineering foundation to build type-safe, production-grade AI systems.”
Understanding vector space math, attention mechanisms, embeddings, and token behavior from first principles.
Building deterministic, production-grade agent runtimes using Pydantic AI, FastAPI, and structured tool registries.
By mastering traditional AI model mechanics while keeping my software engineering discipline at the core, I can build systems that are both mathematically sound and enterprise-ready.
That is the engineer I am building myself to be.