Professor Palsson, thanks for a fascinating article.
I do think that you could create something (at least almost) as powerful as ChatGPT in 1000 AD, though. If my understanding of Textbooks Are All You Need (https://arxiv.org/abs/2306.11644) is correct, you need only about 7 billion tokens (~4.5 billion 6-letter words) of textbook-quality data to create a powerful LLM. If the average textbook has 100,000 6-letter words, that's 45,000 textbooks. That's the output of ~100,000 trained professionals working for 1-3 years.
If the results from the above paper generalize, a lot less than the total number of poets in human history could generate enough training data to create a decent poetry-writing LLM.
You can generate text from these less-powerful-than-ChatGPT LLMs to create training data for a more powerful LLM, your 1000 AD ChatGPT.
You could object that you'd need an army of professionals to invent a 1000 AD LLM, but I'd counter that you'd need such an army to build any complex invention: semiconductor fabs, the Internet, etc. In fact, most of these latter inventions would require more than ~100,000 trained professionals, I think.
Also note that time can make up for a lack of training data. GPT-4 is much more powerful than ChatGPT, even though both were trained on basically the same data, because GPT-4 is more cleverly engineered.
The last thing I need is more economics books, this is as good excuse as any to buy more though.
Professor Palsson, thanks for a fascinating article.
I do think that you could create something (at least almost) as powerful as ChatGPT in 1000 AD, though. If my understanding of Textbooks Are All You Need (https://arxiv.org/abs/2306.11644) is correct, you need only about 7 billion tokens (~4.5 billion 6-letter words) of textbook-quality data to create a powerful LLM. If the average textbook has 100,000 6-letter words, that's 45,000 textbooks. That's the output of ~100,000 trained professionals working for 1-3 years.
If the results from the above paper generalize, a lot less than the total number of poets in human history could generate enough training data to create a decent poetry-writing LLM.
You can generate text from these less-powerful-than-ChatGPT LLMs to create training data for a more powerful LLM, your 1000 AD ChatGPT.
You could object that you'd need an army of professionals to invent a 1000 AD LLM, but I'd counter that you'd need such an army to build any complex invention: semiconductor fabs, the Internet, etc. In fact, most of these latter inventions would require more than ~100,000 trained professionals, I think.
Also note that time can make up for a lack of training data. GPT-4 is much more powerful than ChatGPT, even though both were trained on basically the same data, because GPT-4 is more cleverly engineered.