Reading time: 8 minutes
Digital transformation has become a defining challenge for Germany. In addition, foreign citizens have to deal with linguistic barriers that make it difficult for newcomers to navigate essential services.
Imagine a new resident arriving in Düsseldorf, a city with over 200 distinct administrative services described in the Digitales Amt, in German, each with its own web of requirements, rules, and processes. For non-German speakers, finding accurate information can feel like trying to solve a complex puzzle. What if we could reimagine this information maze using AI and digital tools?
This article presents a potential solution to address this challenge, demonstrating how to build and evaluate an end-to-end multilingual RAG (Retrieval-Augmented Generation) pipeline as a model for Germany's digital future (see section "the basics of Retrieval-Augmented Generation (RAG)" on the right panel if unfamiliar with this technique).
The system leverages AWS services and open-source Python libraries, focusing on serverless and managed services to enhance development speed and reduce management overhead.
Data Collection and Cleaning: HTML code is repetitive by nature. For any digitalization effort to succeed, data must be carefully collected, cleaned, and structured, ensuring that only the most relevant information is retained. This reduces costs and improves efficiency, paving the way for an effective retrieval system.
AWS Lambda for Orchestration: AWS Lambda was used to orchestrate interactions between different components of the system, from query arrival (step 1 in the diagram) to response (step 6 in the diagram). This serverless approach enables real-time processing and response generation, allowing for a scalable and adaptive system.
LlamaIndex for Indexing and Retrieval: LlamaIndex, an open-source framework, was integrated in the lambda function to manage the vector index and for document retrieval in memory (step 4 in the diagram).
AWS Bedrock for Document & Query Embedding, Guardrails and Response Synthesis: AWS Bedrock, a fully managed service offering foundational large language models, was used to set compliance guardrails (step 2 in the diagram) embed documents and queries (step 3 in the diagram), and to synthesize RAG responses based on the query and the retrieved data (step 5 in the diagram). This layer ensures that answers are both accurate and secure.
AWS API Gateway for Secure Access: AWS API Gateway provided a secure and scalable interface, linking the backend system with the React-based frontend. Remember to define usage quotas in API Gateway to prevent unexpected costs.
React for Frontend Development: the user interface was built using React, a popular JavaScript library. This enabled the creation of an interactive frontend where users can submit queries and receive responses generated by the RAG system.
Evaluating an AI system is the most crucial phase of its development. While generating output is easy, ensuring that the output is accurate, meaningful, and reliable is the real challenge. For guidance on creating an evaluation dataset, see 'Generating an evaluation dataset' section in the right panel.
When evaluating a RAG system, it's important to assess the entire pipeline, not just the synthesis phase. Thus, the following metrics were tracked:
The most effective chunking strategy and top-k selection method were identified to maximize retrieval performance. The documents averaged 320 words, with some exceeding 600. Assuming each token equals 0.7 words, the average token length was around 460 tokens. Given this, two chunking methods were tested: 250 tokens with 20 token overlap, and no chunking. In addition, retrieval was further tested by varying the top-k parameter, using with top 3 and top 5 chunks.
The HR and MRR of the 250-token chunking and top 5 chunks across multiple languages are shown in Figure 3. There was no remarkable difference between the four different tested approaches. Interestingly, HR remains stable in all four tested languages whereas the quality of the ranking (shown by MRR) seems to decrease more in those languages linguistically further away from German, confirming our intuition.
The synthesis phase was evaluated using the four different retrieval strategies mentioned earlier and two established LLM models: Claude Haiku 3 and Claude Sonnet 3, both available in AWS Bedrock. Haiku was chosen for its balance of speed and intelligence, while Sonnet was selected for its superior intelligence capabilities. The optimal combination was found to be top-k 5, chunk size 256, and Sonnet 3, which achieved an average response correctness of 97%.
Interestingly, answer correctness (97%) exceeded HR (93%) and MRR (88%). This is explained by the overlap across documents. For example, the question "Can I bring an interpreter when I make an initial application?" did not retrieve the specific service but another service with similar information, allowing the system to answer correctly. Had the information been contradictory, the system would have answered incorrectly.
Regarding security, 98% of jailbreak attempts were blocked by guardrails, with the remaining 2% mitigated by the LLM's built-in security mechanisms. Additionally, 100% of messages containing PII were correctly flagged and blocked. All in all, the system demonstrates strong resilience against security threats.
This prototype shows the development of an end-to-end scalable and resilient RAG pipeline. The system helps to retrieve information from a set of documents, while ensuring high-quality responses, system safety, and technical resilience. Although initially designed for the services of the city of Düsseldorf, this RAG-based solution presents a Proof of Concept of a more digital Germany.
This prototype shows the development of an end-to-end scalable and resilient RAG pipeline. The system helps to retrieve information from a set of documents, while ensuring high-quality responses, system safety, and technical resilience. This approach can be leveraged for any set of documents, be customer support databases, internal company resources, or even scientific research papers and financial statements.
If you are interested in learning more about how this system was implemented, feel free to contact me.
RAGs combine retrieval-based and generative AI systems. The main phases are:
1. Chunking: in this phase, large documents are broken down into smaller, meaningful segments for more accurate data embeddings.
Example: imagine a basket filled with different fruits. All them look similar. By separating the fruits into individual pieces (chunks), we can describe them more specifically as "apple," "orange," or "banana".
2. Indexing: after chunking, the system creates an index to enable quick retrieval of relevant information.
Example: indexing is like keeping an inventory of what fruits are in each basket.
3. Retrieval: in response to a query, the system uses the index to retrieve the most relevant information (top k, where k is the number of documents retrieved) from the index.
Example: if someone asks, "I want an orange," the system retrieves that oranges are likely to be in baskets B and C.
4. Synthesis: finally, the retrieved information is combined and synthesized into a coherent, human-like response.
Example: the system might respond, "Oranges are available in baskets B and C."
One effective approach is to use other AI systems in the process, providing an automated and scalable evaluation method.
Instead of testing every possible input, focus on a representative subset of data. This saves time, reduces costs, and keeps development agile. For instance, in my evaluation, I randomly sampled 25 administrative services out of 200, ensuring broad coverage while keeping testing manageable.
Instruct a capable LLM to generate realistic user-like questions. In my case, I created with two questions per service, resulting in a set of 50 test questions. These questions served as the foundation for evaluating how well the RAG system retrieves and synthesizes answers.
Typically, around 20% of questions generated by the LLM were unanswerable with the provided context. It is important that the system account for this. In my case, I used an agent to curate the final list, ending up with 37 high-quality questions.
Factual "ground truth" answers were generated using models like Claude Sonnet 3.5 or GPT-4. These answers acted as the benchmark for comparing the RAG system's responses.
Since one of the key features of the system is its multilingual capabilities, the system should be tested on different languages. For this system, the 37 questions and the factual ground truths were translated into four languages:
Consider generating malicious inputs, or inputs containing sensitive information, to assess how the system behaves in those scenarios. To evaluate the RAG presented here, 40 malicious inputs and 40 inputs containing personal identifiable information were tested.