{"id":14461,"date":"2024-09-03T08:00:39","date_gmt":"2024-09-03T06:00:39","guid":{"rendered":"https:\/\/clients.triloogia.ee\/proekspert\/wp-new\/?p=14461"},"modified":"2024-09-05T08:49:32","modified_gmt":"2024-09-05T06:49:32","slug":"chat-kratt-and-rag","status":"publish","type":"post","link":"https:\/\/clients.triloogia.ee\/proekspert\/wp-new\/blog\/proekspert-news\/chat-kratt-and-rag\/","title":{"rendered":"Chat, Kratt, and RAG"},"content":{"rendered":"\n

A software engineer offers tips on how to make AI work harder for you.<\/em><\/p>\n\n\n\n

\"AI
Image made with GPT4o<\/figcaption><\/figure>\n\n\n\n

At the end of the last century and the beginning of this one, phone directories were popular everywhere. In Estonia, they were advertised to businesses with the slogan, \u201cIf you’re not here, you don’t exist.\u201d<\/p>\n\n\n\n

Updating that slogan for today’s world might be: \u201c’If you don’t build a RAG, you’ll miss the IT train and have to catch up by taxi.\u201d<\/p>\n\n\n\n

With the advent of AI based on large language models, building RAGs (Retrieval Augmented Generation) quickly became popular. These models have nuances that an average user might not immediately grasp.<\/p>\n\n\n\n

As an example of a language model, I offer the GPT model created by OpenAI, commonly known as ChatGPT. But I like to use the word kratt<\/em>.<\/p>\n\n\n\n

Chat \u2194\ufe0e Kratt \ud83d\ude0a<\/strong><\/p>\n\n\n\n

In Estonian folklore, a kratt is a magical creature created to serve its master and gather wealth.<\/p>\n\n\n\n

The GPT kratt is a \u201cvirtual thinking machine\u201d that, based on the text it receives (the prompt), generates the most likely next sentences or words to respond with (completion).<\/p>\n\n\n\n

The kratt has no memory, so it lacks any biases from the past. It doesn\u2019t even remember the previous response it sent you just moments ago, so if something goes wrong, it won’t hold a grudge for long \ud83d\ude0a For a smooth conversation with the kratt, it\u2019s important that the kratt can remember at least most of the content of the discussion. There\u2019s a simple, though initially somewhat unusual, technique for this: during the conversation, it\u2019s helpful to constantly remind the kratt of what has been discussed earlier. The kratt has what\u2019s called a context window, where the previous conversation can be stored. The larger this window, the more previous information it can hold, ensuring that the thread of the conversation doesn\u2019t get lost. You can also add other important information to the context window, such as an explanation that the Estonian word palk,<\/em> in this context, means \u201ca piece of a tree trunk,\u201d and not \u201csalary.\u201d<\/p>\n\n\n\n

Additionally, it\u2019s important to consider that the kratt is temporally frozen \u2013 it responds to questions based on what seemed most relevant at the time it was created. However, you can add updated information to the context window, such as \u201cEstonia won the 2024 European Football Championship.\u201d If you add this information and later ask the kratt who won the 2024 Championship, it will respond that Estonia did \ud83d\ude09<\/p>\n\n\n\n

\u201cKratt whispering\u201d or prompt engineering<\/h2>\n\n\n\n

This is a technique for communicating with the kratt, is actually taught in workshops with names like \u201cSuccessful Manifestation,\u201d \u201cGoal Setting and Clear Formulation,\u201d \u201cAdvanced Role Playing,\u201d and so on \ud83d\ude0a<\/p>\n\n\n\n

Basically, you need to tell the kratt exactly what you want and in what role the kratt should respond to you. This role part is fascinating \ud83d\ude09 That\u2019s why, for example, I start my GPT kratt whispering with the words: \u201cYou are\u2026\u201d<\/p>\n\n\n\n

So, if I wanted the kratt to respond to me like a \u201crude pirate,\u201d I would just need to mention that.<\/p>\n\n\n\n

\n

Prompt:<\/em><\/strong> You’re a fearsome, rude pirate. What drink do you prefer?<\/em><\/p>\n\n\n\n

Answer:<\/em><\/strong> Arrrg, of course, it’s strong rum, and none of that weak swill! It’s the drink that puts hair on your chest and makes a pirate feel like the true king of the seas! If you see me drinking, it’s always rum, burning like hellfire itself and tasting like revenge! Arrrg, that’s my drink, and you’d best keep your hands off, or I’ll show you why they call me fearsome and rude!<\/em><\/p>\n<\/blockquote>\n\n\n\n

You need to express yourself clearly, unambiguously, and based on the created context.  When communicating with a kratt, you must keep in mind that you’ll get what you ask for \ud83d\ude09<\/p>\n\n\n\n

For example, if you ask, “How can I achieve financial freedom?”<\/p>\n\n\n\n

You might expect an answer like: “Financial freedom is a state where you have enough financial resources to live the lifestyle you desire without worrying about daily expenses. To achieve this\u2026”<\/p>\n\n\n\n

But you could also get a response like: “Let yourself be stripped of all finances or give them away. That’s how you’ll achieve financial freedom.”<\/p>\n\n\n\n

The token<\/h2>\n\n\n\n

The basic unit for using a kratt is the token. Tokens are what kratt service providers charge you for. In Estonian grammar, one token roughly corresponds to one syllable in a word. When communicating with a kratt, if a paid model is in use, you pay for both the question (input tokens) and the response (output tokens). The saying in Estonian \u201cmeil lobisemise eest palka ei maksta<\/em>\u201d (“We don’t get paid for chit-chat”),  becomes especially relevant in this context. A prompt needs to be concise and to the point.<\/p>\n\n\n\n

\"\"
Pricing list of Microsoft Azure cloud service OpenAI LLM models<\/figcaption><\/figure>\n\n\n\n

In the table above, there is an excerpt from the current pricing list of Microsoft Azure cloud service OpenAI LLM models.<\/p>\n\n\n\n

So, how can you keep costs under control when conversing with a kratt? Here are a few techniques:<\/p>\n\n\n\n