{"id":490,"date":"2024-03-03T21:41:38","date_gmt":"2024-03-03T21:41:38","guid":{"rendered":"http:\/\/10.1.7.2:9701\/?p=490"},"modified":"2025-11-24T09:55:40","modified_gmt":"2025-11-24T09:55:40","slug":"damienai-v0-10-v0-32-2-personal-project","status":"publish","type":"post","link":"https:\/\/davidsula.com\/index.php\/490\/damienai-v0-10-v0-32-2-personal-project\/","title":{"rendered":"DamienAI v0.10 \u2013 v0.32.2 Personal Project"},"content":{"rendered":"\n

Note from November 24th, 2025:<\/em>
For those reading this now, this blog post was me diving in cluelessly and doing everything the wrong way. As of present day, DamienAI now outperforms ChatGPT and is useful to me in many ways, having access to many services and automating tasks for me. There are many capabilities to discuss, and I will do so in the upcoming “DamienAI v1.0” blog post.<\/em>
I’m pretty much saying, do not take anything here seriously.<\/em><\/strong><\/p>\n\n\n\n

DamienAI is a personal project I started working on around October 2023. The general aim was to make an AI model that was as similar as possible to ChatGPT<\/a> with the limited resources available to me: a good computer paired with an Nvidia GeForce RTX 3090, no money to fund my project, and a lot of misguided hope.<\/p>\n\n\n\n

On my journey making Damien, I\u2019ve made many updates, and this post will provide the general rundown on where I started and where I am as of March 3rd 2024.<\/p>\n\n\n\n

v0.10: Initial Creation, vast data on whatever I could find easily accessible.<\/h3>\n\n\n\n

At first, I started by coding a basic training script which uses PyTorch<\/a>, and the pre-trained \u201cgpt2-medium\u201d model with the tokenizer that comes along with it (I used the help of GPT4 ironically). This training script would get all the \u201c.txt\u201d files from the \u201cdata\u201d directory, and merge them all into one text file with the titles of each text file being placed above their contents in the combined file. Then this script would tokenize the combined file (make it readable to the model) and train the model on it. Once finished, the script would delete the combined and tokenized files, and end.<\/p>\n\n\n\n

train_model.py<\/summary>\n
import os\nimport pandas as pd\nimport torch\nfrom transformers import GPT2LMHeadModel, GPT2Tokenizer, TextDataset, DataCollatorForLanguageModeling, Trainer, TrainingArguments\n\ndef csvtotext(csvpath):\n    # Read the CSV file\n    df = pd.read_csv(csv_path)\n\n    # Convert the dataframe to a single string\n    # Assuming the CSV has a single column, adjust if it has multiple columns\n    text_data = \"\\n\".join(df.iloc[:, 0].dropna().tolist())\n\n    return text_data\n\ndef train_gpt2():\n    # Set the device to cuda:0\n    device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n    torch.cuda.set_device(device)\n    print(f\"Using device: {device}\")\n\n    # Load pre-trained model and tokenizer\n    model = GPT2LMHeadModel.from_pretrained(\"gpt2-medium\").to(device)\n    tokenizer = GPT2Tokenizer.from_pretrained(\"gpt2-medium\")\n\n    # Get all .txt and .csv files in the data directory\n    data_dir = \".\/data\"\n    txt_files = [f for f in os.listdir(data_dir) if f.endswith('.txt')]\n    csv_files = [f for f in os.listdir(data_dir) if f.endswith('.csv')]\n\n    # Concatenate all text files, using the filename as the subject\n    combined_data = \"\"\n    for txt_file in txt_files:\n        subject = os.path.splitext(txt_file)[0]  # Filename without .txt extension\n        try:\n            with open(os.path.join(data_dir, txt_file), 'r', encoding='utf-8') as f:\n                content = f.read()\n                combined_data += f\"[{subject}] {content}\\n\"\n        except UnicodeDecodeError:\n            print(f\"Error reading {txt_file}. It might not be encoded in UTF-8. Skipping this file.\")\n\n    # Process .csv files\n    for csv_file in csv_files:\n        subject = os.path.splitext(csv_file)[0]  # Filename without .csv extension\n        content = csv_to_text(os.path.join(data_dir, csv_file))\n        combined_data += f\"[{subject}] {content}\\n\"\n\n    # Save the combined data to a temporary file\n    temp_file_path = os.path.join(data_dir, \"combined_training_data.txt\")\n    with open(temp_file_path, 'w', encoding='utf-8') as f:\n        f.write(combined_data)\n\n    # Prepare training dataset\n    train_dataset = TextDataset(\n        tokenizer=tokenizer,\n        file_path=temp_file_path,\n        block_size=128\n    )\n\n    # Data collator handles batching\n    data_collator = DataCollatorForLanguageModeling(\n        tokenizer=tokenizer,\n        mlm=False\n    )\n\n    # Define training arguments\n    training_args = TrainingArguments(\n        output_dir=\".\/trained_model\",\n        overwrite_output_dir=True,\n        num_train_epochs=1,\n        per_device_train_batch_size=32,\n        save_steps=10_000,\n        save_total_limit=2,\n    )\n\n    # Create Trainer instance\n    trainer = Trainer(\n        model=model,\n        args=training_args,\n        data_collator=data_collator,\n        train_dataset=train_dataset,\n    )\n\n    # Train the model\n    trainer.train()\n\n    # Delete the cached file\n    if os.path.exists(temp_file_path):\n        os.remove(temp_file_path)\n\n    # Delete the cached tokenizer file\n    tokenizer_cache_file = os.path.join(data_dir, \"cached_lm_GPT2Tokenizer_128_combined_training_data.txt\")\n    if os.path.exists(tokenizer_cache_file):\n        os.remove(tokenizer_cache_file)\n\n    # Explicitly save the model and its configuration\n    trainer.save_model(\".\/trained_model\")\n\nif __name == \"__main\":\n    train_gpt2()\n    input(\"Press Enter to continue...\")<\/code><\/pre>\n<\/details>\n\n\n\n
The data I initially trained the AI on was only 35.5 Megabytes, and it involved 90 Wikipedia documents which were scraped using a simple script which prompted for a link to a Wikipedia article and scraped it to a \u201c.txt\u201d file.<\/p>\n\n\n\n
WikipediaScraper.py<\/summary>\n
# WikipediaScraper.py\n\nimport requests\nfrom bs4 import BeautifulSoup\nimport tkinter as tk\nfrom tkinter import simpledialog, messagebox\n\ndef scrape_wikipedia_page(url):\n    response = requests.get(url)\n\n    if response.status_code != 200:\n        print(f\"Failed to retrieve the webpage. Status code: {response.status_code}\")\n        return None\n\n    soup = BeautifulSoup(response.content, 'html.parser')\n    content_div = soup.find('div', {'id': 'mw-content-text'})\n    for table in content_div.find_all('table'):\n        table.decompose()\n\n    page_text = content_div.get_text()\n    return page_text\n\ndef save_content_to_file(page_name, content):\n    filename = f\"{page_name}_Wikipedia.txt\"\n    with open(filename, \"w\", encoding=\"utf-8\") as file:\n        file.write(content)\n\ndef main():\n    root = tk.Tk()\n    root.withdraw()  # Hide the main window\n\n    while True:\n        url = simpledialog.askstring(\"Input\", \"Please enter the Wikipedia URL:\")\n\n# If the user presses Cancel or closes the window, exit the loop\n        if not url:\n            break\n\n        content = scrape_wikipedia_page(url)\n        if content:\n            page_name = url.split(\"\/\")[-1]\n            save_content_to_file(page_name, content)\n            messagebox.showinfo(\"Success\", f\"Content saved to {page_name}Wikipedia.txt\")\n        else:\n            messagebox.showerror(\"Error\", \"Failed to scrape the content.\")\n\nif name == \"_main\":\n    main()<\/code><\/pre>\n<\/details>\n\n\n\n
The Wikipedia data totalled up to 10MB on its own. The rest of the data came from publicly accessible datasets, such as the subreddit Cornell, which came in a 10MB text file, some Joe Rogan Experience transcripts which were auto-generated by YouTube (and very poor), and a couple of other minuscule random sources.<\/p>\n\n\n\n
The result was a really bad AI model that simply did not work. While training the AI model, there is a factor presented to the user called \u201ctrain_loss\u201d. The train loss is essentially how well the AI model understands the data it is being trained on, measured through the difference between the guesses that the AI makes as to what comes next in the data, and what actually is in the data. For reference, when the AI model reaches a train loss of 0.2, it can be stated that the AI understands the data it is being given very well. Anything around 0.5 means that the AI is generally understanding the data. Anything above 1.5 signals that the AI is dysfunctional or broken. DamienAI started, after being trained for the first time, with a train loss of over 3\u2026 This is entirely due to a lack of data.<\/p>\n\n\n\n
This resulted in awful responses that made no sense from DamienAI. Essentially what it was doing was continuing from what I gave it. So if I messaged it a word that was mentioned in its training data, it would find whatever words often came after that word, and return them to me. Here are some examples. (I mentioned Rishi Sunak and Joe Rogan as it was trained on Wikipedia articles on those people.)<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
v0.20: Fine-tuning<\/h3>\n\n\n\n
On the OpenAI website, they publish documentation on the methods they use to train their ChatGPT. They train their AI in two steps. The first step is to train a blank slate on vast amounts of data; to OpenAI this involves over half a terabyte of text data. The next step is \u201cfine-tuning\u201d. This is where they custom-write their own data consisting of queries followed by responses, and train the AI on this data so that it learns the behaviour of an assistant or chatbot.<\/p>\n\n\n\n
Having read this, I custom-wrote my own queries and responses to feed into my model. This was a 14KB \u201c.txt\u201d file which I trained DamienAI on several times.<\/p>\n\n\n\n
\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n<\/div>\n\n\n\n
Whilst these outputs were good, there were still a lot of errors, most commonly being the output being the same as the input, or the repetition of the same word over and over:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
These errors were still primarily because I had trained the model on such a small amount of data.<\/p>\n\n\n\n
v0.25: Feedback<\/h3>\n\n\n\n
The next step I thought of was to create a feedback system for the model so that if a good response is given, I can tell Damien that the response is good so that he is encouraged to repeat that output. My method for this was to code it so that after every response, the user was prompted to say whether or not the response was good. These results would be outputted to a \u201c.csv\u201d file, and the outputs rated \u201cgood\u201d would be returned to the model for re-training, in hopes that these outputs would occur more often as a result.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
v0.30: Fresh restart with Wikipedia Data.<\/h3>\n\n\n\n
I decided to take a step back and restart from scratch. I improved my Wikipedia scraping script so that now it would not only scrape Wikipedia articles, but it would scrape any hyperlinks in the articles it is scraping, as Wikipedia tends to use hyperlinks over key terms as these would direct the viewer to the Wikipedia article on that term. This resulted in around 20-30 Wikipedia articles being scraped per link that I provided to the script. As a result, this new set of data totalled up to 953MB of data \u2013 a huge improvement, although still nowhere near enough for the AI model as the train loss came out as 2.9. This was still a lot better than what was well over 3, although it did also show that I need to improve my methods of harvesting data massively.<\/p>\n\n\n\n
I\u2019m not going to show any responses from this model, as I hadn\u2019t fine-tuned it so DamienAI would just be responding with whatever words it found associated in the Wikipedia articles with the words I provided.<\/p>\n\n\n\n
v0.31<\/h3>\n\n\n\n
In this update, I simply added a \u201ccache\u201d folder so that the training script could dump its combined data file and its tokenized data file whilst it uses them for training, as well as providing it with a place to store \u201ccheckpoints\u201d, which are saves of the model midway through its training, just in case something happens to the system and it stops abruptly.<\/p>\n\n\n\n
v0.32: Organising<\/h3>\n\n\n\n
I began to rework how my data is stored, and I also began to seek data from more sources. My data directory went from being a dump of loads of unorganized text files to a clean organized directory which had several paths to different types of data.<\/p>\n\n\n\n
Before:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
After:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
This data is very useful as it is just general conversation which is what I primarily want DamienAI to understand. Each podcast has about 200KB of text data, which is very good.<\/p>\n\n\n\n
Additionally, I found out that Project Gutenberg<\/a> provides free E-Books which you can just download in plain text, so this also provides large sums of text data. I managed to download 694MB of data from Project Gutenberg (964 E-Books, although they have over 70,000 on the website so downloading more is a must).<\/p>\n\n\n\n
BBC News also became a source of text data, although very small, as each news article has around 5KB of data.<\/p>\n\n\n\n
I did not finish this version, as I wanted to add Multiple GPU compatibility first so that I could use Google Cloud VMs which provide several powerful GPUs per VM.<\/p>\n\n\n\n
v0.32.2: Multiple GPUs + more data<\/h3>\n\n\n\n
I updated my \u201ctrain_model.py\u201d script to use any available Nvidia Cuda capable GPUs on the system for training, as I was hoping to use Google Cloud VMs, although it is just not possible to use. I tried using Google Cloud VMs for a week and I was not capable of getting a single one throughout that week as they were never available. I tried all types of GPUs available, I also tried lowering the amount of GPUs I am using to a single one. I never got a chance to use it.<\/p>\n\n\n\n
train_model.py<\/summary>\n
# train_model.py\n\nimport os\nimport pandas as pd\nimport torch\nfrom transformers import GPT2LMHeadModel, GPT2Tokenizer, TextDataset, DataCollatorForLanguageModeling, Trainer, TrainingArguments\nimport concurrent.futures\n\ndef processfile(filepath, basedir):\n    file_name = os.path.basename(file_path)\n    file_name_without_ext = os.path.splitext(file_name)[0]\n\n    relative_path = os.path.relpath(file_path, start=base_dir)\n    path_components = relative_path.split(os.sep)\n    path_tags = path_components[:-1]\n\n    # Adjusted to format directory path with ' > ' and place content on a new line\n    tag = \" > \".join(path_tags)\n\n    try:\n        with open(file_path, 'r', encoding='utf-8') as f:\n            content = f.read().strip()\n            # Adjusted to place the first line of content below the title\n            return f\"[{tag} > {file_name_without_ext}]\\n{content}\\n\\n\"\n    except UnicodeDecodeError:\n        print(f\"Error reading {file_path}. It might not be encoded in UTF-8. Skipping this file.\")\n        return \"\"\n\ndef train_gpt2():\n    if torch.cuda.is_available():\n        device = torch.device(\"cuda\")\n        print(f\"Using device: {device}, with {torch.cuda.device_count()} GPUs\")\n    else:\n        print(\"CUDA is not available. Training on CPU.\")\n        device = torch.device(\"cpu\")\n\n    model = GPT2LMHeadModel.from_pretrained(\"gpt2-medium\")\n    tokenizer = GPT2Tokenizer.from_pretrained(\"gpt2-medium\")\n\n    # Enable Data Parallelism for multi-GPU training\n    if torch.cuda.device_count() > 1:\n        model = torch.nn.DataParallel(model)\n\n    model.to(device)\n\n    base_dir = \".\/data\"\n    all_files = [os.path.join(root, file) for root, dirs, files in os.walk(base_dir) for file in files if file.endswith(('.txt', '.csv'))]\n\n    with concurrent.futures.ProcessPoolExecutor() as executor:\n        combined_data_list = list(executor.map(process_file, all_files, [base_dir]*len(all_files)))\n\n    combined_data = \"\".join(combined_data_list)\n\n    cache_dir = \".\/cache\"\n    os.makedirs(cache_dir, exist_ok=True)\n\n    temp_file_path = os.path.join(cache_dir, \"combined_training_data.txt\")\n    with open(temp_file_path, 'w', encoding='utf-8') as f:\n        f.write(combined_data)\n\n    train_dataset = TextDataset(tokenizer=tokenizer, file_path=temp_file_path, block_size=128)\n    data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)\n\n    training_args = TrainingArguments(\n        output_dir=\".\/trained_model\",\n        overwrite_output_dir=True,\n        num_train_epochs=1,\n        per_device_train_batch_size=36 \/\/ torch.cuda.device_count(),\n        save_steps=20_000,\n        save_total_limit=20,\n    )\n\n    trainer = Trainer(\n        model=model,\n        args=training_args,\n        data_collator=data_collator,\n        train_dataset=train_dataset,\n    )\n\n    trainer.train()\n    trainer.save_model(\".\/trained_model\")\n\nif __name == \"__main\":\n    train_gpt2()<\/code><\/pre>\n<\/details>\n\n\n\n
In the end, I gave up on using Google Cloud as it was just not possible.<\/p>\n\n\n\n
I did work on getting a bit more data and finally managed to pass the 1 gigabyte mark. The training data I used on v0.32.2 totalled up to 1.25GB. This included Wikipedia articles, Scholarpedia articles (Wikipedia alternative), documentation for open-source projects, documentation for coding languages, BBC News articles, and more.<\/p>\n\n\n\n
In the end, I had to train the model on my computer using my RTX 3090. I had left it to train overnight (It took 11 hours) and the result was a train loss of 2.9. This goes to show I need much greater methods of collecting data, as 1.25GB is still nowhere remotely close to enough.<\/p>\n\n\n\n
Training Screenshot<\/summary>\n
<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
<\/p>\n<\/details>\n\n\n\n
My next steps for v0.33 are to fine-tune this model just to see how it goes, and I will document my process and the results. For v0.4, I will aim to collect a lot more data.<\/p>\n","protected":false},"excerpt":{"rendered":"
DamienAI is a personal project I started working on around October 2023, with the goal of making an AI model like ChatGPT.<\/p>\n","protected":false},"author":1,"featured_media":854,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[],"class_list":["post-490","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blogs"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/davidsula.com\/index.php\/wp-json\/wp\/v2\/posts\/490","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/davidsula.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/davidsula.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/davidsula.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/davidsula.com\/index.php\/wp-json\/wp\/v2\/comments?post=490"}],"version-history":[{"count":2,"href":"https:\/\/davidsula.com\/index.php\/wp-json\/wp\/v2\/posts\/490\/revisions"}],"predecessor-version":[{"id":1177,"href":"https:\/\/davidsula.com\/index.php\/wp-json\/wp\/v2\/posts\/490\/revisions\/1177"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/davidsula.com\/index.php\/wp-json\/wp\/v2\/media\/854"}],"wp:attachment":[{"href":"https:\/\/davidsula.com\/index.php\/wp-json\/wp\/v2\/media?parent=490"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/davidsula.com\/index.php\/wp-json\/wp\/v2\/categories?post=490"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/davidsula.com\/index.php\/wp-json\/wp\/v2\/tags?post=490"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}