Scott was doing great work with the database, but the hungry beast that was ONElist demanded more people to work on it. So I took Ethan Kan out to lunch. Ethan was the second smartest engineer I knew (after Scott). He had started working at Diba, the startup that Scott and I were at, right when Diba was bought by Sun Microsystems. Bad timing, I suppose, for Ethan.
Luckily for us, Ethan decided to join ONElist. So now ONElist was up to 3 people, all moonlighting on ONElist, while still working full time jobs. Let me tell you, that is not a fun way to live.
Around this time, customer support was becoming a big problem. It was getting to the point that there would be 200 support emails every night, and 500 emails on weekends. I was going insane trying to answer them. So, I recruited Carisa Cannan, my girlfriend, to help answer all the messages. Nice boyfriend, eh? We would trade off answering emails, but she ended up answering most of them throughout the rest of the summer. I was well on my way to insanity before she started helping out.
August 7 – ONElist Upgrade Report
For this upgrade, we were moving the archives off of the original ONElist PC onto another system. Even at this point, there was a lot of data to move. Another sleepless weekend.
From: ONElist Tech Support
Date: Fri Aug 7, 1998 8:28am
Subject: ONElist Upgrade Report
We are continuing the system-wide upgrade of ONElist. We recently added additional mail capacity to ensure that email will not be delayed. The next step in the upgrade will happen over two nights, Friday August 7th and Saturday August 8th. For this part of the upgrade we will have to take the ONElist system off-line for approximately 8 hours each night, starting at 6pm Pacific Time. Email that is sent during this time will be queued up and delivered once the system is back on-line.
This is the second to last step of the upgrade process. We appreciate your patience during our upgrades.
Thanks for using ONElist.
The ONElist Team
August 27-29 – Oracle Upgrade Weekend
Finally, Scott and Ethan had put together an Oracle database system for ONElist. We had no idea if it’d really work, although they had tested it a bit. The old database system was doing so badly, that we decided to rush the new system into production. We would fix anything that broke as we went along. Or at least that was the theory.
From: ONElist Tech Support
Date: Thu Aug 27, 1998 11:12am
Subject: Upgrade Notice
We will be doing the final part of our upgrade Friday evening, starting at 7Pm Pacific Time. The upgrade is scheduled to last approximately 8 hours. Email sent during that time will be queued up and delivered when the system is back on-line.
We appreciate your patience during our upgrade process, and we look forward to being able to bring you many exciting new features soon.
Eight hours was wildly optimistic, but we had no way of telling how long the move to Oracle was going to take. So that Friday, we left work early and began the upgrade. The upgrade process involved Scott and I, and we each worked from our homes, calling each other every couple of hours. That weekend, both my brother and my girlfriend were out of town. So I was able to obsess over ONElist unimpeded.
After 8 hours, the upgrade wasn’t complete, but by this point there was no going back. We had to finish it. Around 5Am Saturday morning, everything was copied over into Oracle and we were ready to go. Scott fired up the system, and things seemed to work. But we noticed that things weren’t quite as fast as they should be. Being very tired, we gave up for the night and tried to get some sleep.
It was apparent that Saturday that things were not right. The new database was, in fact, running more slowly than the old database. Which was kind of opposite of what we were going for. This had to be fixed. After much head pounding, we determined that the way that we were talking with the database was the cause of the slow down. We quickly set about creating a new server system that would fix the problem, but it required a lot of work.
Eventually, around 3pm Sunday afternoon, the new server was ready. It had been tested for all of 5 minutes, but we had to put it into production. The ONElist web site had basically been unusable all weekend, because things were so slow. In a sane world, when you write a new software program, you carefully design it. And when it’s completed, you test it a lot to ensure that there are no bugs. But in our insane world, we didn’t have the time to do that. So we put the new server on-line. And miracle of miracles, it worked. Instantly, the web site was fast, email delivery worked again, birds started singing, and a rainbow descended upon my house. I went to bed and dreamed of donuts.