Server for RAG consultant: on-premise or rental, foreign and Russian LLMs

Running a RAG consultant “on a demo server” is usually not difficult. The complexity begins at the production stage: when you need to maintain SLA, differentiate access, store logs, update indexes without downtime and control the cost of inference.

This is why in AI implementation projects, not only the choice of model is important, but also the right infrastructure. In most cases, the decision is made between two scenarios:

• on-premise (within the company);
• infrastructure rental (cloud/VPS/DC in the Russian Federation or abroad).

If you need a complete turnkey project, including architecture, deployment and integration, this is closed on the page AI consultant with RAG.

When to choose on-premise

On-premise is more often chosen by companies where the following are critical:

• protection of personal and commercial data;
• internal information security policies and access audit;
• industry compliance requirements;
• minimizing external dependencies.

Pros of this approach:

• full control over data and logs;
• flexible configuration of the network and role model;
• predictable behavior under high load.

Limitations:

• higher entry threshold for infrastructure;
• a maintenance process is needed (DevOps/MLOps);
• scaling requires planning resources in advance.

When is it more profitable to rent capacity?

Rental infrastructure is suitable when speed and flexibility are important:

• quick pilot launch;
• push-button scaling as load increases;
• less capital costs at the start.

Pros:

• short time-to-market;
• convenient experimentation with multiple LLMs;
• It's easier to run test environments for teams.

Risks that need to be covered:

• data placement policy;
• jurisdiction and terms of the provider;
• perimeter and access key control;
• budget stability during sudden peaks in requests.

Minimum infrastructure requirements

The exact configuration depends on the number of users, the context length, and the selected model. But for orientation it is convenient to use basic levels:

1. Pilot (up to 30-50 active users): CPU + moderate RAM, limited document pool.
2. Department/direction: dedicated vector index, stable inference circuit, monitoring.
3. Corporate outline: separate dev/stage/prod environments, fault tolerance, backups and DR plan.

In production, it is important to include not only hardware, but also operational processes:

• latency/throughput/quality metrics;
• alerts and logging;
• version control of indexes and prompts;
• regulations for incidents and rollback.

Choice of LLM: foreign, Russian or hybrid

In most enterprise projects, a hybrid approach works:

• foreign models are used for complex generation and reasoning tasks;
• Russian models - for local scenarios, regulatory restrictions and cost optimization;
• Query routing selects a model based on task type.

Key idea: there is no need to “put one model on everything.” It is much more stable to divide scenarios according to the following criteria:

• criticality of accuracy;
• data sensitivity;
• acceptable delay;
• cost of response.

RAG circuit safety

The information security issues here are systemic, not specific. Minimum set:

• RBAC/ABAC by roles and sources;
• data encryption during storage and transmission;
• sanitization of input requests;
• protection against prompt injection and data exfiltration;
• masking sensitive entities in logs.

For internal implementations, it is useful to immediately describe the threat model: what data cannot be released and under what conditions the response should be blocked.

Architecture that works most often

A practically sustainable option for medium-sized businesses:

1. Data sources: CRM, ticket database, Wiki, documents.
2. ETL layer with update scheduling and deduplication.
3. Vector index + keyword search (hybrid).
4. LLM-router for selecting a model for the scenario.
5. API layer and UI channels (internal portal, chat, helpdesk).
6. A set of quality metrics and operational monitoring.

If you need a fast pilot, you can start with a reduced contour, and then increase reliability as the business effect is confirmed.

How to estimate TCO before launch

Before starting, it’s worth calculating not only the cost of development, but also the Total Cost of Ownership:

• infrastructure (servers/cloud, storage, network);
• inference(tokens/requests/load);
• support for indexes and knowledge base quality;
• DevOps/MLOps support;
• licenses and integration connectors.

A good TCO estimate reduces the risk of a situation where the pilot looks cheap and the operating budget gets out of hand.

Where does it relate to CRM and internal processes

In practice, the maximum effect of RAG is manifested in conjunction with sales and service processes: quick responses to managers, tips on regulations, faster processing of applications. Such a circuit can be associated with development of custom CRM or connect when implementation of a ready-made CRM. Automation example - case AI2Media: AI platform and working scenarios.

FAQ

Is it possible to start without a GPU?
For a limited pilot, yes, but as the load increases and scenarios become more complex, a separate inference circuit is usually required.

What to choose first: on-premise or cloud?
If safety and regulation are critical - on-premise. If the priority is launch speed and hypothesis testing - the cloud.

Do I need to connect several LLMs at once?
Not necessarily, but it is better to design the architecture so that you can add a second model without refactoring the entire system.