Sam Hartman

I wrote about how I’m exploring the role of AI in human connection and intimacy. The first part of that journey has been all about learning the software and tools for approaching large language models.

The biggest thing I wish I had known going in was not to focus on the traditional cloud providers. I was struggling until I found runpod.io. I kind of assumed that if you were willing to pay for it and had the money, you could go to Amazon on or google or whatever and get the compute resources you needed. Not so much. Google completely rejected my request to have the maximum number of GPUs I could run raised above a limit of 0. “Go talk to your sales representative.” And of course no sales representative was willing to waste their time on me. But I did eventually find some of the smaller AI-specific clouds.

I intentionally wanted to run software myself. Everyone has various fine-tuning and training APIs as well as APIs for inference. I thought I’d gain a much better understanding if I wrote my own code. That definitely ended up being true. I started by understanding PyTorch and the role of optimizers, gradient descent and what a model is. Then I focused on Transformers and that ecosystem, including Accelerate, tokenizers, generation and training.

I’m really impressed with the Hugging Face ecosystem. A lot of academic software is very purpose built and is hard to reuse and customize. But the hub strikes an amazing balance between providing abstractions for common interfaces like consuming a model or datasets without getting in the way of hacking on models or evolving the models.

I had a great time, and after a number of false starts, succeeded in customizing Llama2 to explore some of the questions on my mind. I’ll talk about what I accomplished and learned in the next post.