Experiments - AI, Rust, GPU etc.

A few times a year, I have an idea that makes me think: this is something I want to sit down for and fully develop. One of those was Katmai - The Virtual Office, which was also commercially successful. On this page, a selection of some experiments.

DawnSearch (2024) - A Decentralized AI Search Engine in Rust

DawnSearch was an experiment based on a number of ideas:

  1. Google functions worse than it used to.
  2. The reliance on a small number of major search engines is concerning.
  3. AI might be able to contribute something here.
  4. It would be great to run everything in a distributed way.

Additionally, I wanted to do a larger project in Rust to gain more experience with it.

In the end, this proof of concept turned out reasonably well. I loaded the data via the CommonCrawl project using a custom parser for the file format. Then I calculated an embedding (with BERT) of the text (via Candle), and stored it in a vector database. The application itself consists of a crawler and a web UI (server-side, entirely in Rust). I had ambitious plans for clustering, but at this point, it only includes discovery and forwarding queries and responses to other nodes. All of this is highly optimized and uses raw UDP packets.

All in all, a valuable learning project. It made it clear to me that Rust is quite overkill if you want to build regular software. It's certainly suited for parsers and the like, but everything around that is so much harder compared to, say, Java or TypeScript. I personally prefer Java.

The code is available on GitHub (AGPLv3),

Evolving on the GPU (2023) - Simulation of Life and Diversity

This experiment had two goals:

  1. I wanted to simulate "life" and evolution in an interesting way.
  2. I wanted to gain more experience with running computations/simulations on the GPU.

This experiment was a success in the sense that I managed to simulate a large grid with enormous performance, and that the modeling of "life" had some quite interesting outcomes. It became clear that evolution is not so simple and that various "islands" and conditions are needed to create diversity and avoid getting stuck in a local optimum.

The code is not currently available (but feel free to send me an email if you'd like me to upload it to GitHub).