Strawberry I-don't-know, and an agent implementation

Reinforcement with 'i-don'know'

 OpenAI has just released strawberry/o1, and we are now very close to AGI.

It is pretty much using the same technique that I have already outlined in a previous blogpost.

Create synthetic data with trains of thought, where each thought is a Q/A. This turns the problem into a reinforcement problem, where the evaluation of each answer can be scored.

One crucial thing they unfortunately didn't do, was to mark answers that are wrong as i-don't-know.

Then the algorithm would be to train with a small nerual net the first round, find all answers that are wrong, and then train again with a bigger neural net. Scoring wise a wrong answer is worse than saying i-don't-know, which is then worse the correct answer.

Since final neural net is bigger than the one that was original trained, it should have a high chance of answering i-don't-know, when it needs to.


Agent

Having a strong LLM is just one part of AGI. The AGI also needs to be able to take actions.

I have been implemented a wrapper in the following way:

 

The overall idea is you start a conversation with the AI, and the AI runs in the background to solve your problem. While the AI is running it can AI can ask questions to the user. And the user can also add additional information to the conversation at any point. It feels very much like having a human assistant.

To do this it consist of multiple programs, that talk to each other through the console.  Each program has a thread where it can take input, that runs independent of the work the program does. This means that information can pass down/up through the stack, independently of the work that is being done. The work does not need to be interrupted, since it writes its progress into info memory. When the process than gets done with a step, it can view new information, and react on it.

The AI have some interesting ideas, that maybe can be inferred by the above picture. But one interesting one, is that the context window is kept as small as possible at any time. This means that a task has no information when it starts, but have the ability to search and add additional information from other tasks or elsewhere. And only the minimum amount of information needed to succeed, is added to the task.

Another thing is that task are retried if the success criteria fails. This means:

If the tasks are correct, and the success criteria is correctly evaluated, then overall plan will always succeed if all steps can succeed once.

The last thing is the response improver, which are very much tied to the OpenAI strawberry. The first time a request is done to an LLM, it is done to a very small an fast LLM. But the response improver runs in the background, and improves responses in a order where it make the most sense. It could be to improve the overall plan, or steps that are currently failing.

That means you original task you start as a human, never actually completes. But you can see the latest response, and how good the AI think the response is, and how much work has gone into it so far.

Kommentarer

Send en kommentar

Populære opslag fra denne blog

Older blog posts

These where originally posted on https://omnia.dk/hard_ai Hard AI Lean Fuglsang, 2023-12-31 Current LLMs like ChatGPT are very impressive, but are also limited when giving complex task. So it can not do the following: "Give investment advice" "Write a research paper" "Improve the AI located in /home/me/HardAI/CurrentBest" But what if we can super charge the LLM to do the above tasks? The idea is that to do a task, we don't do a single request to the LLM, but maybe thousands or millions. We use the LLM input as working memory - all non-pretrained information that is needed for the request. And instead of given the final respone, the LLM will return the next action that should be taken. The result of the action is then summarized into the work memory - again by using the LLM. To enhance the performance, we can add an additional validation step that mitigates the problems of the LLM halucinating, being imprecise or have wrong alignment. An...
Læs mere

Week 7

Google has released the Gemini LLM - which has a context window of 1 million tokens. Google blog They also write that it can analyse it all with what they call a 'needle in a haystack' test. And more importantly it can do 'in context' learning, and actually using the information in the context window for complex tasks. This means that the concept of a CoreLLM is much closer to being useful. It looks like the context window just keep growing, so there is a balance how small the CoreLLM should be, and how much should be put in the context window. This also means the AI singularity is getting closer - the day when the AI improves the code it is built of. Even if the AI improves it self, it is not guaranteed that it can improve itself in all areas. But in that case it is important to remember that it can easily be nudged by humans. If you have an idea how to improve the algorithm that the AI can not figure out it self you can just put that information in the short term or l...
Læs mere

Week 12 - part 2

I saw this tweet by Yan Lecun that LLM can not think abstract in a visual way https://twitter.com/ylecun/status/1769778223842226286  What he fails to realize, or maybe exactly what he realised is that you should not make an LLM answer a question directly. Instead it shuold always be wrapper: "What is the strategy for answering this prompt: {X}, having these tools at your disposal". And then the list of actions that can be done. Think of it if you got some liquid and was asked what it was. It would be dangourus to just taste it, and give an answer. Instead we will analyze it - find the pH value and what other tools we can use to find out what it is. And then take that knowledge, and give a result. So the job of the LLM is to do this analysis and continuously improve the prompt (short term memory) with more and better information. And of course also improve the state for the tools available (long term memory, the programmed world model). The prompt can both contain text, images...
Læs mere