AI BUILD PLAYBOOK
SWAG
AI Engineering · Playbook

How I Build AI Projects
From Scratch

A repeatable, product-minded process for turning a blank repo into a real AI application — the same approach used building production AI/ML systems at Amazon.

🎯 Learn 📣 Demonstrate 👥 Real Users Example: ShelfScanner

scroll to start ↓

The Workflow at a Glance

10 Steps, One Process

01Know WHY you're building 02Pick a problem you understand 03Scope it before you build 04Work with AI the right way 05Set up the foundations 06MVP that answers unknowns 07Build in phases 08Evaluate on two layers 09Monitor it 10Deploy & present it

the order matters more than the labels

01 · Motivation

Why Build Projects?

01

Teach a Skill

Building a project makes you learn a capability for real — not just follow a tutorial.

02

Demonstrate It

Show employers you can build something robust and use good judgment while doing it.

03

Become Experience

With real users it turns into résumé experience — or a product you could monetize.

  • Three different goals, three different bars. A learning project can be simple; the same project on a résumé needs robustness and visible judgment.
02 · Ideation

What Makes a Good Project?

  • Start with a problem you understand — then work backwards to the technology.
  • Don't learn about RAG / agents / a new model and go hunting for a place to jam it in.
  • Source problems from your own life, people around you, or a community (Reddit / Discord).
  • The best portfolio projects aren't always the most technically impressive — they're the ones you have a personal connection with.
  • Hiring managers look for enthusiasm + decision-making — which only shows if you came up with the idea yourself.
SCOPE
03 · Scoping

Scope It Before You Build

  • Turn a vague idea into something decision-ready: who it's for, constraints, input/output data, success metrics.
  • Add the practical ones early too: latency goals and a budget.
  • Use a scoping template — but most of it stays blank on purpose.
  • The question people skip: does AI even make sense here?
  • If normal software solves it just as well, a model just adds cost and unpredictability.

a record of what you believed before you started

04 · Working with AI

Work With AI the Right Way

  • Use spec-driven development — the assistant drafts a proposal before it implements anything.
  • Review that proposal with a very critical eye: fix assumptions you don't agree with.
  • Keep decisions in files, not chat history, so you remember what and why.
  • If you outsource your thinking, you can't defend the decisions in an interview — because you didn't make them.
Docs directories Steering / rules files Installed skills Proposal → review → build
05 · Foundations

Set Up the Boring Parts

🗄️

Data Layer

You'll work with data. Example uses Supabase — Postgres + file storage in one place, generous free tier.

🔑

Secrets & Env

All API keys in a .env file, correctly gitignored. Manage the Python env with uv.

  • Create project → install CLI → grab API keys → initialize. Match every tool to what scoping already identified.
MVP
06 · The MVP

The MVP Answers Questions

  • An MVP is not a crappy version of the whole app.
  • It's the smallest thing that answers your biggest unknowns.
  • Two project-killers to test first: can a vision model read book spines? · can an LLM give useful recommendations?
  • ShelfScanner's first MVP: a command-line script — photo in, titles read, five recommendations out.
  • Find out before spending two weeks on a pretty UI.
07 · Experiments

Treat It Like a Real Experiment

Which model ran Which prompt What it returned How long it took Rough cost Did it fail
  • Log every model call — don't go by vibes.
  • Keep shelf-reading and recommendation stages separate: the best OCR model isn't necessarily the best recommender.
  • Separate stages → compare each problem cleanly, and swap one model without touching everything else.
08 · Test Data

Start With a Small Labeled Set

  • Start with five photos you already have — not to prove it works, but to ask is this promising enough to continue?
  • Write labels against a consistent rubric.
  • Rules don't need to be perfect — but they must be applied the same way, or grading is random and useless.
  • Start with a router like OpenRouter to test several providers through one interface, fast.

boring labeling saves you later

RESULTS
09 · Findings

What the MVP Taught Me

  • Vision: Sonnet & Gemini Flash found essentially all readable books without inventing any — smaller models missed titles and hallucinated books that weren't there.
  • Cost: far less interesting than expected — good models sat inside budget.
  • Latency: the real problem — 10–12s just to read a shelf vs. a ~15s target for the whole app.
  • Recommendations: cheaper GPT-5.4 mini matched my choices best; the nicest-sounding explanations scored worse.
  • Privacy: source images carried GPS metadata that shouldn't be stored.
10 · Restraint

Leave Out What the Problem Doesn't Need

✕ RAG

No retrieval problem — books come from the current shelf.

✕ Agents

The workflow is known ahead of time: read, verify, recommend five.

✕ Fine-tuning

Off-the-shelf models are good enough as-is.

  • Also out of V1 scope: logins, accounts, purchase links.
  • No extra points for an architecture diagram stuffed with RAG, three agents, a fine-tuned model and six databases.
  • If the problem doesn't need it, leaving it out is the better decision.
11 · Phasing

Build Riskiest First

  • Break work into phases that each leave something working end-to-end, one addition at a time.
  • Order matters more than the phases: deal with risky parts while they can still change the design.
  • No point polishing a recommendation screen before you know the recommendations are good.
  • Rough deadlines per phase prevent scope creep.
100+
Commits in the rebuild
8
Weeks (≈6 building)
10–15
Hrs / week
<½
Spent actually coding
12 · Evaluation

Two Layers of Correctness

Layer 1

Normal Software Tests

  • Test pieces with a known correct behavior
  • Valid uploads pass; bad inputs rejected; data hits the right user
  • Full browser flow tests — parts can pass while the product breaks
  • Runs in CI/CD via GitHub Actions on every push & PR
Layer 2

AI Evaluation Set

  • Rerun on any change to models, prompts or preprocessing
  • Labeled photos + deliberately degraded versions (blur, glare, rotation, low-res)
  • Catches regressions & hallucination — nothing crashes when behavior degrades
  • Compare prompt versions systematically — only possible with versioned prompts

code can be perfectly valid while behavior gets worse

13 · Monitoring

Monitor It — Log Everything

  • Reuse the model-call logs from experiments to power a simple dashboard: model, latency, cost, prompt version, failures.
  • Optionally add alerts for the unexpected.
  • Feed repeated real-world failures back into the eval set, then fix and keep checking.
  • Collect feedback signals in the product — saves, "not for me."
  • Add optimizations like caching only for a reason you can defend — never "because production apps have caches."
14 · Deployment

Ship It Off Your Laptop

  • Vercel + GitHub: connect the repo, a couple of clicks, deployed.
  • Every change auto-deploys once pre-deployment checks pass.
  • A custom domain made it feel real — shelfscanner.io.
  • Now usable the way it was imagined: walk up, snap a photo, get recommendations.
  • Honest caveat: it's a web app, not native — "good enough to show the process."
15 · Career

Present It Well, Then Get Users

  • The engineering is invisible from the outside — a bare link wastes it.
  • Write a real README: trade-offs, functionality, small architecture diagrams.
  • Get real users — even a couple dozen teach you a lot and give great interview stories.
  • Enough users ≈ professional experience: the project moves from "Projects" to "Experience."
  • This is how you beat the need-experience-to-get-experience catch-22.
TAKEAWAYS
Recap

Key Takeaways

01Decide why you're building — learning, demonstrating, or real users.
02Start from a problem you understand; work backwards to the tech.
03Scope before building — and always ask if AI is even the right tool.
04Go spec-driven so you make the decisions and can defend them.
05Make the MVP answer your biggest unknowns — not a mini copy of the app.
06Log every model call and keep stages separate.
07Build riskiest parts first; give phases deadlines.
08Evaluate on two layers with versioned prompts.
09Monitor in production; let failures feed the eval set.
10Ship, present, get real users — turn a project into experience.
Final Word

Don't Copy ShelfScanner.
Copy the Process.

The project is a framework, not a template. The value comes from making your own decisions — and being able to explain why.

now go build something of your own ✦

01 / 19