RAG vs Fine-Tuning vs Prompting – A Decision Map for Startup Founders

Signiance Technologies > blog > AWS > RAG vs Fine-Tuning vs Prompting – A Decision Map for Startup Founders
Signiance Cloud Architecture Illustration 1

Name the failure first, then pick the cheapest fix that clears your quality bar on Amazon Bedrock

In this article

  1. Start With Prompt Engineering and Prompt Caching
  2. Add RAG When the Problem Is Missing Knowledge
  3. Fine-Tune When the Problem Is Behaviour or Cost at Scale
  4. The RAG vs Fine-Tuning Decision Map on One Page
  5. How Signiance helps

Most founders ask this question in the wrong order. They hear that a competitor fine-tuned a model and wonder if they should too, before they have tested a well-structured prompt against fifty real examples. Choosing between RAG vs fine-tuning vs prompting is not about which technique sounds most advanced. It is about what is actually wrong with your model’s answers today.

There are only three kinds of problems a language model can have inside your product:

  1. It does not understand the task. The instructions are unclear, the format drifts or it skips steps.
  2. It does not know the facts. Your pricing, your docs, a customer’s contract or yesterday’s ticket history are not in its training data.
  3. It knows and understands, but it does not hold up at scale. It is too slow or too expensive at your volume, or it will not behave consistently no matter how you phrase the prompt.

Prompting fixes the first. Retrieval augmented generation (RAG) fixes the second. Fine-tuning, in its several forms, fixes the third. Most production systems end up using all three in layers, but the order in which you add them decides how much time and money you spend. Here is the decision map we use with startup teams building on Amazon Bedrock.

Start With Prompt Engineering and Prompt Caching

Prompting is the cheapest experiment you can run. Changing a system prompt takes minutes, needs no data pipeline and is fully reversible. Before you reach for anything else:

  • Write a system prompt that states the role, the task, the output format and two or three worked examples.
  • Collect 30 to 50 real inputs from your users or support inbox, along with what a good answer looks like.
  • Run every prompt change against that set and score it. A spreadsheet is fine at this stage.

If quality is acceptable but a long prompt is driving up cost or latency, add prompt caching before anything else. On Amazon Bedrock, prompt caching lets you mark cache checkpoints so a stable prefix (tools, system prompt, long reference text) is reused across calls instead of being reprocessed. AWS positions it as a way to reduce inference latency and input token costs. A few details matter in practice:

  • The cached prefix has a minimum size. Depending on the model, minimums currently range from 512 to 4,096 tokens.
  • The default cache TTL is 5 minutes and resets on every hit. Some models also support a 1-hour TTL, set with "ttl": "1h" on the cache point.
  • Order matters. Bedrock processes tools, then system, then messages, so a change early in the prompt invalidates everything cached after it. Keep volatile content at the end.
  • On some models cache writes cost more than standard input tokens, so caching pays off when the same prefix is hit repeatedly, not on one-off calls.

A minimal Converse request with a cached system prompt looks like this:

{
  "system": [
    {"text": "You are the support assistant for Acme. Rules and examples follow..."},
    {"cachePoint": {"type": "default"}}
  ],
  "messages": [
    {"role": "user", "content": [{"text": "How do I reset SSO for my team?"}]}
  ]
}

Stay in this lane while your failures are about instructions and format. Move on when your evaluation set shows the model inventing facts it could not possibly know.

Add RAG When the Problem Is Missing Knowledge

If the model gives confident but wrong answers about your product, or the right answer depends on data that changes every week, no prompt will fix it. That is a knowledge problem, and RAG is the right tool. Retrieval keeps facts outside the model, so updating an answer means updating a document, not retraining anything.

Amazon Bedrock Knowledge Bases is the managed path. It searches your data at query time and passes the relevant chunks to the model, with citations back to the source. For a startup, the useful parts are:

  • Connectors. The managed option connects to Amazon S3, SharePoint, Confluence, Google Drive, OneDrive and a web crawler, so you are not writing an ingestion job for each source.
  • Permissions. Managed knowledge bases support ACL-based document permission filtering (the web crawler is the exception), which matters the moment two customers share one assistant.
  • Two retrieval APIs. Retrieve returns chunks so your own code or agent decides what to do with them. RetrieveAndGenerate returns a grounded answer with citations in one call.
  • Structured data. For questions that live in tables, knowledge bases can turn a natural language question into SQL through the GenerateQuery API.
  • GraphRAG. For questions that span relationships across documents, there is a managed GraphRAG option built on Amazon Neptune Analytics. Read its limits first: Amazon S3 is the only supported data source, for example.

The vector store is your main cost decision. Amazon S3 Vectors can sit behind a knowledge base, and AWS said at launch that it can reduce the total cost of uploading, storing and querying vectors by up to 90 percent. The S3 docs cite sub-second cold queries and warm queries as low as 100 milliseconds. The trade-off is that it offers semantic search only, with no hybrid keyword search, and custom metadata per vector is capped when used with knowledge bases. If your users search for SKUs, error codes or exact names, test hybrid search on a store that supports it, such as Amazon OpenSearch Serverless, before you commit.

One more lesson from real projects: most “RAG does not work” complaints are retrieval problems, not model problems. Check chunk size, metadata filters and whether the right document ever reaches the prompt before you blame the model.

Fine-Tune When the Problem Is Behaviour or Cost at Scale

Fine-tuning changes the model’s weights. It is the right answer when you need consistent behaviour that prompting cannot hold, a very specific style or output schema, or a smaller model that performs like a larger one on your narrow task. It is the wrong answer for teaching facts that change, because every update means another training run.

Amazon Bedrock currently offers three model customization methods, and the data you already have usually decides which one fits:

| Method | What you need | Best for | |—|—|—| | Supervised fine-tuning | Labeled input and output examples | Fixed formats, tone, domain phrasing | | Reinforcement fine-tuning | Prompts plus a reward function (rules or an AI judge) | Tasks where you can score answers but cannot write the perfect one | | Distillation | Prompts for your use case | A large model’s quality from a smaller, cheaper model |

Reinforcement fine-tuning is the newest option. In February 2026 AWS extended it to open-weight models, starting with Qwen3 32B and gpt-oss-20b. Rewards come from rule-based graders or AI judges, with custom grading logic in AWS Lambda, and the trained model can be called through Bedrock’s OpenAI-compatible APIs without a separate deployment step.

Distillation suits a startup that already has a prompt working well on a large model and now wants similar results for less. Bedrock uses the larger teacher model to generate responses to your prompts, then fine-tunes the smaller student on them. The synthetic data generation is billed at the teacher model’s on-demand rates, so budget for it.

Serving cost is where founders get surprised, so check how your customized model will run before you train it. Bedrock supports on-demand inference for some custom models, which means you pay per use instead of provisioning capacity. Today that covers custom versions of Amazon Nova Micro, Nova Lite, Nova 2 Lite and Nova Pro, plus Meta Llama 3.3 70B Instruct, in specific US Regions. AWS prices on-demand inference for custom Nova models the same as base Nova inference, and storage for each custom model is billed monthly. Other paths can require provisioned capacity, which changes the economics completely for an early-stage product.

The RAG vs Fine-Tuning Decision Map on One Page

Run through these questions in order and stop at the first yes:

  1. Is the model misunderstanding the task or format? Improve the prompt and add examples. Add prompt caching if the prompt is long and reused.
  2. Does the right answer depend on private or changing data? Add RAG with Bedrock Knowledge Bases, and pick the vector store based on whether you need hybrid search.
  3. Do answers need citations or per-user permissions? RAG again. Fine-tuned weights cannot cite a source or respect document ACLs.
  4. Is behaviour still inconsistent after a strong prompt and good retrieval? Supervised or reinforcement fine-tuning, depending on whether you have labeled examples or only a way to score answers.
  5. Is quality fine but cost or latency too high at your volume? Distil into a smaller model, and confirm on-demand serving is available for it.

Two rules hold across all five. First, build the evaluation set before you change anything, because every option above is a guess until you measure it. Second, expect to combine them. A common production stack is a fine-tuned or distilled small model for tone and format, RAG for facts and a cached system prompt on top.

How Signiance helps

Signiance is an AWS Partner working with startups on Generative AI on AWS. We help teams build the evaluation set first, test prompting, RAG and model customization against it on Amazon Bedrock, and choose the cheapest option that clears the quality bar. Then we ship it with the observability and cost controls to keep it there.

If you are not sure which of the three your product actually needs, book a free AI Native readiness assessment or a free consultation at signiance.com. We will map your use case to the right approach in a single session.