Memcache-based circular buffer for Google App Engine

The problem: AppEngine's datastore is slow and uses a lot of cpu.

One solution: store non mission-critical data in memcache only. Your app might not have any data that you can be ok with losing, but there are a lot of good examples for games: chat messages, console messages, status updates... it's ok if these things get lost for some reason and we aren't interested in keeping them around for a long time anyway.

Here is some code to get you started with a circular buffer. Basically we have 10 memcache keys for data and one key that is used as an index (iter). The last ten items stored are available in order from most recent to least recent, and if a new item is added it replaces the oldest item while still being returned at the front of the list thanks to the sorting.

If you don't have other things you don't mind being kicked out of cache you can modify this code to just keep increasing the iter value instead of performing mod 10 such that old keys are not overwritten. This should save you some cpu time. Eventually the old items will be kicked out of cache automatically.

Anyway, here is the code:

from google.appengine.api import memcache

def addToBuffer(bufferData):
  loc = memcache.incr("myBufferIter")
  if loc == None:
    memcache.set("theIter", "0")
    loc = 0
  memcache.set(key="myBufferData%d" % (loc % 10), value=text)

def getBufferData():
  keys = []
  try:
    iter = int(memcache.get(key="myBufferIter"))
    for i in range (0, 10):
      keys.append('%d' % i)
    results = memcache.get_multi(keys, "myBufferData")

    # You can omit this part if you don't care about the order of the data
    sortedResults = []
    for i in range (iter%10, 0, -1) + range(9, iter%10, -1):
      sortedResults.append(results[str(i)])
    return sortedResults
  except:
    return None

Early lessons about multiplayer game design from Conquer-on-Contact

Right now I'm working hard on a new release of Conquer-on-Contact that addresses the biggest problems the previous version had. I learned a few important lessons from the initial release that are sure to be old hat for experienced game designers, but they were new to me.

1. Most people do not read instructions. A lot of people do, but enough won't that the interface needs to be as intuitive as possible.
2. If a game allows it, some percentage of people will play for hours on end. This is why most multiplayer online games give you a fixed number of turns per day. Bandwidth isn't cheap so I need to limit use too, but I'm trying to discourage people from playing too much with game mechanics aside from strict turn limiting. This may turn out to be too difficult.
3. Test, test, test. For the new release of Conquer-on-Contact I've written scripts that simulate player actions and I use them to simulate hundreds of simultaneously active players. It gives my poor quad-core a workout, but it's the only reliable way to catch certain obscure bugs.
4. Multiplay has to be taken into account. Most online games try to limit multiplay by saying they'll ban all your accounts if they find you playing more than one. In practice this is impossible without having someone watch player actions very closely, and even then it's mostly a wash. One way to address the multiplay problem is by making it such that other players can only really hurt each other rather than help each other, but you have to be careful here too because alliances can be a lot of fun.
5. Multithreadding is hard. Especially on a platform like App Engine where there's basically infinite parallelism and limited high-speed access to shared state. Unless you're a whiz when it comes to distributed processing, it's not a bad strategy to record user intentions in queues and then use a single-threadded handler to carry out those requests in sequence periodically. You can't officially do background tasks on App Engine yet, but you can simulate them easily enough with a cron process running on another machine. There are a lot fewer edge cases to worry about when you're processing requests in a single thread rather than trying to deal with them in a massively parallel fashion as they come in, and an added bonus is that you can respond to user queries with a brief acknowledgement very quickly.
6. High scores, rankings, and other "carrots" are important. If you present a high score list, the top high score will be the goal that most players are trying to achieve. I'd like to add more intermediate carrots but there probably won't be time before the next release.

Conquer-on-Contact has been a great learning experience for me. I can hardly recommend this route enough if you want to learn what makes games fun or gain more experience with distributed programming. Be sure to check out the new version this Sunday!

Optimizing your productivity with multiple screens and virtual desktops

I wrote an answer over on stackoverflow that was voted up so I thought I'd elaborate on the idea and post it here.

This is going to be old hat for you long-time linux gurus, but for ages I was stuck in the paradigm of one desktop, a taskbar, and window management. I blame my extensive use of Macs and Apple's lack of support for virtual desktops when I was growing up. Today, with linux, compiz, and the era of $10/GB DRAM, you really have no excuse not to embrace multiple desktops.

The idea is that instead of doing window management, you just leave all your windows open and arranged all the time. Never minimize again. Just switch between virtual desktops when you need to "context switch" from one project or workflow to another.

So for a web development project you might have one virtual desktop with:

• Monitor 1: VIM
• Monitor 2: Firefox
• Monitor 3: xterms, other misc stuff

On another virtual desktop you might have Photoshop and GIMP and some other image tools. And on another desktop you might have another instance of firefox for fun and irc and your chat sessions.

Here's what my set of 9 desktops looks like right now (all together it represents 10080*3150 pixels):

Once you've tried this way of working you'll feel positively clausterphobic when you're forced onto a system that doesn't support virtual desktops.

Have Ubuntu 8.10? Have Nvidia? Update your drivers!

I was getting really annoyed with my slow Firefox and Compiz performance in Ubuntu 8.10. I have a quad core and a 8800GTS! It should be extremely fast. Then I noticed that the nvidia drivers Ubuntu installed automatically were positively ancient! They were in the 172.xx range when 180.29 is out now. Do yourself a favor and update to Nvidia's latest drivers right now.

You'll need to remove the old drivers first:

sudo apt-get remove nvidia*

Then download the new drivers from Nvidia's website and install. Since you need to close your x session to do this, I recommend you ssh into your box from another machine so that you can get a nice terminal to run nvidia's installer (It seems to have some problems if you run it from the console; I got flashing text and other nonsense). Once you've modprobed the new driver, get ready to enjoy the 2x or greater boost in screen redraw speed and true 60fps for your cube if your card is fast enough!

Sensible Eating Guidelines, Or Why You Shouldn't Believe What You Read

Image via Wikipedia

Some years ago I decided I wanted to eat more healthily, but I realized I didn't actually know what 'healthy' meant. I started studying the best I knew how, which was to start with Google and go from there. Nothing could have prepared me for the mess I was about to get into.
Here are a few things I learned:

• I don't trust the food industry. Their objective is to sell their products, and they'll tell us whatever their focus groups say we want to hear.
• I don't trust the government. The FDA and the USDA and all these agencies do help consumers in some ways, but ultimately they are all subject to industry pressures through a variety of channels.
• I don't trust people who write books about food because they are usually more interested in selling their books than in presenting all sides of a story.
• I don't trust the opinions of random people because, for the most part, they just tend to repeat what the above entities say. Independent thought is rare.

You shouldn't trust what I'm saying here, either. I have my own agenda: I want you to read my blog and like it and tell all your friends about it. So consider me as just one data point in a vast network of opinions that you need to consider. Food in modern society is a topic complex enough to devote a career to, but it's worthy of some study from you because, hey, your eating habits are probably more important than any other factor in determining how long you live and what your overall quality of life is.
These are some of the guidelines I personally find useful:

Be practical.

You can go overboard with anything. Like everything else in life, the quality of your food choices is a trade-off. You have to choose between time, money, enjoyment, potential longevity, and other factors.

Less is more.

Studies on animals indicate that calorie restricted diets promote longevity. Odds are good this applies to humans as well. I personally feel better and have more energy when I'm at a lean weight, so this is a no brainer.

When practical, choose plant over animal products.

Poisons tend to accumulate as you go up the food chain.

Choose only low glycemic-load carbohydrates.

Insulin spikes seem to cause diabetes, put you at risk for heart disease, and make you sleepy. Why bother? Stick to the time-release carbs.

Avoid trans-fat.

It's associated with heart disease.

Avoid genetically or otherwise modified foods.

We evolved eating certain types of food. Modifying those foods is not necessarily a bad thing, but it's somewhat like modifying a running software program: without extensive testing you don't know what the results are going to be. And you are kidding yourself if you think these companies do extensive tests.

Take it all with a grain of salt, and remember this is not advice. It's what works for me.

Does music increase or decrease productivity?

A Google query for "music work decreases productivity" mostly turns up hits that suggest listening to music while you work will increase your productivity.

My experience is mostly the opposite. I suppose if I were doing something I found very boring that music would help keep me entertained and perhaps more productive, but most of the work I do requires 100% of my focus. Distractions of any kind—yes, even music—pollute my ability to concentrate and slow me down. In noisy environments, earplugs are far better at reducing the impact of the distractions than covering them with music is.

On the other hand, working with music is clearly more pleasant. I like to sing along when songs have vocals and it's enjoyable for me. But I don't confuse enjoyment with productivity.