Week 12 - part 2

I saw this tweet by Yan Lecun that LLM can not think abstract in a visual way

https://twitter.com/ylecun/status/1769778223842226286 

What he fails to realize, or maybe exactly what he realised is that you should not make an LLM answer a question directly.


Instead it shuold always be wrapper: "What is the strategy for answering this prompt: {X}, having these tools at your disposal". And then the list of actions that can be done.


Think of it if you got some liquid and was asked what it was. It would be dangourus to just taste it, and give an answer.

Instead we will analyze it - find the pH value and what other tools we can use to find out what it is. And then take that knowledge, and give a result.

So the job of the LLM is to do this analysis and continuously improve the prompt (short term memory) with more and better information. And of course also improve the state for the tools available (long term memory, the programmed world model). The prompt can both contain text, images and videos (and even other formats if needed). So the LLM can generate an image that it then can use to do a better analysis.

On important adavantage is that the AI can also much easier explain how it got to the answer.


To summarize the purpose of the LLM is not to immediatly give a final result, but to take the current prompt, add some information, and then use that again.



Kommentarer

Send en kommentar

Populære opslag fra denne blog

Older blog posts

These where originally posted on https://omnia.dk/hard_ai Hard AI Lean Fuglsang, 2023-12-31 Current LLMs like ChatGPT are very impressive, but are also limited when giving complex task. So it can not do the following: "Give investment advice" "Write a research paper" "Improve the AI located in /home/me/HardAI/CurrentBest" But what if we can super charge the LLM to do the above tasks? The idea is that to do a task, we don't do a single request to the LLM, but maybe thousands or millions. We use the LLM input as working memory - all non-pretrained information that is needed for the request. And instead of given the final respone, the LLM will return the next action that should be taken. The result of the action is then summarized into the work memory - again by using the LLM. To enhance the performance, we can add an additional validation step that mitigates the problems of the LLM halucinating, being imprecise or have wrong alignment. An...
Læs mere

Week 7

Google has released the Gemini LLM - which has a context window of 1 million tokens. Google blog They also write that it can analyse it all with what they call a 'needle in a haystack' test. And more importantly it can do 'in context' learning, and actually using the information in the context window for complex tasks. This means that the concept of a CoreLLM is much closer to being useful. It looks like the context window just keep growing, so there is a balance how small the CoreLLM should be, and how much should be put in the context window. This also means the AI singularity is getting closer - the day when the AI improves the code it is built of. Even if the AI improves it self, it is not guaranteed that it can improve itself in all areas. But in that case it is important to remember that it can easily be nudged by humans. If you have an idea how to improve the algorithm that the AI can not figure out it self you can just put that information in the short term or l...
Læs mere