The current state of the art from a server capacity perspective is cooperative multitasking and queues. The basic notion is that you should not have to pay for all the context switches in the operating system and the CPU when you don't want/need them. If you design your application to yield everytime it is waiting for something to return then you gain MASSIVE performance improvement both in terms of speed and in terms of not requring huge amounts of memory per simultaneous client. See e.g. The C10K Problem.
In python, your options are async core (aka Medusa) or Twisted.. Medusa is being fecklessly maintained while twisted is in active development so the choice is obviously twisted. If I were working in Java, I might choose Seda.
So having chosen twisted you want to create a src directory. In the src directoy, create a directory with your application name (this is going to be a module name directory or package directory if you are in java). Add the src directory to your compiler or script language loadpath.
In that directory, create a main.py that takes a config and runs your site given your framwork. For Twisted, this looks something like this:
#!/bin/env python
from twisted.internet import app
from twisted.web import static, server,script,vhost
def start(config):
root = static.File(config.fileBase)
root.indices=['index.rpy']
root.processors = {'.rpy': script.ResourceScript}
root.ignoreExt(".rpy")
rootroot = NameVirtualHost()
rootroot.addHost(config.host,root)
application = app.Application('web')
application.listenTCP(config.port, server.Site(rootroot))
application.run()
import sys
if __name__=="__main__":
configPath=sys.argv[1]
execfile(configPath)
main(__name__)
This script allows you to start the site either with config.py as a commandline argument or from another script by passing a config as an argument to start. It also runs the site as a virtual host so that unless someone knows the name of the file your dev site is inaccessible to nosy people on your LAN even if you are exposing the port to the world. Remember to edit your hosts file to get the local name to resolve correctly.
Some friends of mine may know what I am actually working on, but the details are irrelevant here. What this topic is about is how I am developing this project. I spend a few hours on the #twisted irc channel getting a lot of help and want to give back to the community so here goes. How to make a massively cool web site on a tight budget.
1. make a directory for your project on your development computer
2. Build a prototype in HTML that has all the functional options you may want to provide. Show it to some friends and get positive feedback about all the power you can provide.
3. Start thinking about the complexity of specifying and using all those features and shove all those files into a subdirectory of your project called proto
4. check the directory into revision control. I am using subversion on my latptop to a reposiotry also on my laptop.
5. make a directory called site and make the entire HTML for your site again thinking carefullly about how much work it will take for the user to understand what you are offering and whether anyone really cares to have each bit of functionality in version 1. You want to trim to the bare minimum at this stage.
6. Make a directory called src and a directory called test.
7. choose python to develop your application because python programmers are more likely to be smart and saavy and culturally in sync with you than most other languages (because it is a little obscure and a little mainstream and very powerful).
8. Put a config.py file in your test directory. It wll be used by both your test runner and your site runner. config.py should start off looking like this:
fileBase="../site"
scheme="http"
host="myhost"
port=8080
9. Create test.py and start.py in your tests direcotry which both import config.py. start.py starts the website and test.py runs tests against the website. Here is my start.py:
import sys
sys.path.append("../src")
import cospend.main as main
import config
main.start(config)
10. Using pyunit (the unittest module), write tests of the most basic functionality that you intend to implement (e.g. authorization) in test.py. test.py should end with unittest.main(). You may want to create an HTTP client module to automate some of the HTTP stuff inherent in all of these tests. (I did.)
Next up. Choosing a site development model to satisfy the tests and other constraints.
In answer to Islamist concern about maintaing religion in the face of economic growth comes a new study that finds that people who attend religious services regularly build more wealth than those who fail to do so and that their are wide disparaties in wealth production accross religions.
The study suggests that different religions train their adherent in different wealth building behavior and that attending religions services furnishes greater opportunities for social networking. I would add that religions endow their adherents with additional social capital that they can use to cooperate and achieve public goods more efficiently than they could as unaffiliated individuals. For more on why this social capital is important see the end of the enlightenment..
After the Accelerating Change Conference, John Smart and I were debating the merits of spending on better bandwidth vs spending on more CPU power. He noted that bandwidth efficiency is growing more slowly than compute power or storage and argued it was natural. I said that the money spent on the next chip fab would be better spent buying out TV broadcasters and using their spectrum for bits. There are regulatory reasons why that is difficult, but Kevin Werbach shows the value of doing so and how to get there.
Went to this last night. No discussion of how many different social networking variants there actually are or what the opportunities are in the space. No discussion of tacit vs articulated social networks and the benefits of each. No discussion comparing friendster to blogrolls or about the distinction between articulated meta-data and extended personality as expressed through one's blog content. Oh well. better luck next time.
Kevin Werbach recommends looking at Me++ The Cyborg Self and the Networked City. From its web site:
Thus, Mitchell proposes, the "trial separation" of bits (the elementary unit of information) and atoms (the elementary unit of matter) is over. With increasing frequency, events in physical space reflect events in cyberspace, and vice versa; digital information can, for example, direct the movement of an aircraft or a robot arm. In Me++ Mitchell examines the effects of wireless linkage, global interconnection, miniaturization, and portability on our bodies, our clothing, our architecture, our cities, and our uses of space and time. Computer viruses, cascading power outages, terrorist infiltration of transportation networks, and cellphone conversations in the streets are symptoms of a dramatic new urban condition--that of ubiquitous, inescapable network interconnectivity. He argues that a world governed less and less by boundaries and more and more by connections requires us to reimagine and reconstruct our environment and to reconsider the ethical foundations of design, engineering, and planning practice.
Thats right. The experiment of the enlightenment that our minds are separate from our bodies has come to an end.
The enlightenment is failing on all fronts these days. We need to come up with a replacement fast.
Vinod does a great job describing what the Islamists are fightingfor. He quotes a recent Al Queada published book saying:
...The goal of democracy, according to Al-Ayyeri, is to "make Muslims love this world, forget the next world and abandon jihad." If established in any Muslim country for a reasonably long time, democracy could lead to economic prosperity, which, in turn, would make Muslims "reluctant to die in martyrdom" in defense of their faith.
Question for western folks: Is Al Queada correct that economic prosperity and democracy cause secularism?
Given that the most secular cultures are the most infertile cultures (see last post), does this place limits on the overall rate of economic growth for human cultures?
Is economic growth sustainable in the face of declining population?
Some people suggest we can solve the declining growth problem with immigration, but we then face either the cost of acculturating and educating the immigrants and integrating them into our culture or we face an alien population that does not really replace the children lost to infertility.
Can we find ways to grow fast that don't cause secularism and infertility?
Vinod then quotes Islamist author Sajjad Khan as saying:
...The Western viewpoint in life divorces societal actions in this life from the hereafter and therefore ignores any accountability after death. Therefore Western politicians judge actions they do in this life according to their own criteria usually the pursuit of material benefit, i.e. utilitarianism.
In this Khan is correct. So far, the best mechanism for internalizing public good producing behavior that we know of is religion. Given that child production is a public good, are the Islamists right to be alarmed about the threat Western Culture poses to their future generations?
Update/Abstract: The Enlightenment belief in the supremacy of individual reason is a memetic pathogen causing infertility in populations infected by it. Child bearing is a public good but a private cost. Game theory tells us that individual reason actually prevents acquisition of public goods (avoid tragedy of the commons, solve prisoner's dilemma, etc). Acquisition of public goods requires either social norms that require irrational behavior (typically motivated by fear of god(s)) or central government (typically motivated by fear of humans). Enlightenment thinkers typically favors the later. However both economists and computer scientists recognize the former is more efficient (less overhead for economists, more emergent/more software for computer scientists. However the facts are now in. Enlightenment culture is failing to reproduce biologically (most obviously in Europe and less obviously but still visibly among Enlightened Americans). The Enlightenment's phenomenal memetic success is unsustainable. Evolution favors populations that breed. Unless the heirs to the Enlightenment can find other ways to acquire public goods, the survivors of the Enlightenment and the heirs to all its technology will be religious communities that can accept and integrate new technologies while still retaining their integrity. Over the short term innovation may slow compared to the Enlightenment. However it may be the only way forward.
Growth of the human population appears to have started to decline is expected to go negative shortly. By and large the populations most affected by this trend are those that are creating and adopting new technology most quickly. Since technology/innovation is *fundamentally* about the more efficient/non-zero-sum use of existing resources (see prior few posts), we must ask what is the cause of negative human population growth. Given that we have no obvious ecosystem competitor* and individual human lifespans are actually increasing we have only two choices here:
1. Advances in information technology described by Kurzweil and Moore entirely epiphenomenal and we are actually facing a decline in our ability to innovate to alleviate real resource constraints
2. Accelerating change is associated with some pathogen that causes human infertility or is itself the pathogen.
Enough evidence was presented at the conference to reject #1 (see my last few posts).
The leaves us with #2 and forces us to inquire whether all of us technology folks are a problem. Since there is also no evidence of declining fertility rates among couples that choose to reproduce, we are looking at some form of memetic pathogen as the cause of infertility. Less formally, something is causing/allowing rich educated intelligent women to choose to produce less than two children each over the course of their lifetimes. I believe that "something" is the Enlightenment notion that human reason can be used to solve all problems. Enlightenment thought encourages individuals to think for themselves rather than default to traditional/religious values. Enlightenment thought encourages women to abandon their traditional gender roles to advance their own goals and implicitly elevates goals of the mind over goals of the body -- the production of ideas for use by enlightened thinkers. It encourages woman towards memetic rather than sexual reproduction and it encourages men to choose women based on their thinking/conversational ability rather than their reproductive ability.
Enlightenment thinking certainly stirred the pot and motivated its adherents to explore a much larger volume of idea space much faster than any prior ideology. However, from an economic game theoretic non-zero sum perspective and from a John Smart Accelerationist perspective the enlightenment premise that individual reason is the solution to all problems is flat-out wrong.
Economists refer to it as the problem of externalities and of procuring public goods. They also refer to it as the Tragedy of the Commons. Game theorists refer to it as he Prisoner's Dilemma. It is the fact that there are many contexts in which if everyone behaves rationally they all do worse than if they had all behaved irrationally. Farmers overgraze the shared grassland. Because there is "no honor among theives," prisoners will confess rather than risk being the one left out in the cold.
Accelerationists should recognize that uncoordinated individual behavior is a sign of insufficient computational complexity/energy-density. Improvements require a sacrifice of individual freedom/rational-self-interest to participate in in larger meta-organisms. The history of accelerating change is actually one of progressive aggregation of smaller units for the benefits of the order of larger units. Quanta to particles, particles to atoms, atoms to molecules, molecules to cells, cells to multicellular organisms, organisms to packs/herds/tribes, tribes to agricultural communities, and all the way up to nation states today. The change is simply one of maximizing the value of each innovation at larger and larger scales. Strong accelerationsists believe we are accelerating towards cosmic scales. Perhaps we are, but, in the interim, we have to solve the public goods/individidualism problem.
So what does this economics and accelerationist stuff have to do with fertility and reproduction? If you read my last post on the value of population. You will notice that the value of an additional member of the population is likely to be distributed accross the population as a whole. The problem is that the the cost of producing and training this member falls much more heavily on its mother. In economics language, having children has positive externalities that cannot be captured by their producers resulting in a shortage as compared to their theoretic optimum. What is worse is that each decision not to have children increases the relative costs for the women that do. Having children is, therefore, also a public good.
Enlightenment folks typically either try to deny the existence of these sorts of problems (some actually advocate negative population growth!) or, when they can no longer do so, look to various forms of government to coerce people into acquiring public goods. In the case of fertility policy, this coercize force takes the form of higher taxes and sudsidies for child production. The alert reader will recognize that coercize government as an admission of the failure of the pure reason premise of the enlightenment. The economics oriented reader will note the massive inefficiencies associated with government coercion and monopoly rents. The accelerationist reader will note that coercize force is a hardware solution to what should be a software problem (programming coordination behavior). The humanist will just be alarmed at the prospect of population implosion in the near future.
In other words, we should recognize that the era of the Enlightenment is coming to an end. It will be succeeded by cultures that fundamentally recognize individual Embodiment in locations and populations. Embodiment ideologies recognize explicitly individual participate in communities at varying levels and have institutional structures to program coordination behavior into members rather than coerce them. This programming means that idea processing happens at the community level rather than the individual level. It means following community norms even when it appears individually irrational (not defecting in prisoner's dilemma and not overgrazing public goods). It means programming women to have children even when it is against their apparent individual rational self-interest. It means building cultural immune systems to alien memes that could disrupt this programming and it is already happening. The most succesful historical practioners of embodiment ideology have been the members of various religions. Modern Orthodox Jews manage to reproduce themselves while participating actively in Enlightenment culture. Modern Islam has managed to reproduce itself in opposition to Enlightenment culture. If you read radical Islamist literature, Jihadi leaders have an explcit agenda of overtaking the heirs to the enlightenment simply by outbreeding them and then outvoting them.
As heirs to the dying enlightenment culture, we may need to decide which is more valuable, the next generation of chips or the next generation of people. Is it more important to program people or computers?
Update 20030918: I now realize that a shorter statement of the below is that non-zero-sum populations are networks and that Metcalfe's and Reed's law holds. The value of the next member of the population grows either linearly with the size of the preexisting population (Metcalfe) or exponentially with the size of the existing population (Reed).
Robert Wright spoke the next morning about the history of non-zero sum games and how the history of humanity is one of acheiving more and more complex and valuable zero sum games from tribe hunting behavior to agricultural communities to city states to countries to (he hopes) a single global civilization.
Wright presented only the sunny side of non-zero-sumness. The darker side, glossed over in his presentation, is that the the history of humananity is actually a combination of non-zero sum interaction in cases where such interaction is easy and killing/genocide where it is not. The story is one of a brutal evolutionary selection process in favor of genes and cultures that support non-zero sum interaction. The choice of cultivating a non-zero-sum relationship with another party or consuming it (kill it and takes its goods) is a hardcore economic investment decision reflecting the expected value of the possible future relationship and the extent one discounts future value as compared to present value. This investment decision happens at all levels, scaling from the micro-level involving relations between individuals all the way up to nation-states and cultures. In a resource constrained world, those who get this decision right reproduce, those who don't die off. Given that non-zero-sumness is fundamentally about creating new inventions and arrangements, we can establish an equivalence between intelligence and non-zero-sumnesss and say quite explicitly that, in the presence of resource constraints, evolution is selecting humans and human aggregates for intelligence/creativity in solving resource constraint problems.
Wright also talked briefly about the notion that larger populations were better because they were more likely to produce great innovations. But following the economics above, other things being equal, human populations will grow only non-zero sum members and will kill off the rest. The relevant economic terms are decreasing marginal utility and increasing marginal costs. Populations grow until the net marginal value of the next individual to the existing population is zero (growth may mean killing less useful members of the population to make room for new more useful members but that dynamic is beyond the scope here). In other words, all members of the population innovate enough to justify their existence (though the value of those innovations may exhibit a power-law distribution as is visible in the distribution of web traffic and links) so economics/evolutionary selection guarantees that more people are always good for the population as a whole and makes it richer (and if you don't think so, you are obviously a
zero-sum individual scheduled for elimination when the time comes! :-) ).
The afternoon presentations were dominated by a debate about whether the improbability of us all being here today implies anything about whether the universe was designed by an intelligent creator. Personally, I think the universe was created to produce me. I'm sufficiently confident in my views that I went outside to chat with other conference participants.
Note: I'm being flip here, but I do think there is something fundamentally interesting about the interaction between religion and accelerating change/complexity. I'll cover it later when I get to John Smart's talk.
The Accelerating Change conference last weekend began with an ill-formed debate between Ilkka Tuomi and Ray Kurzweil about the selection bias in data Kurzweil uses to "prove" Moore's law. Arguing selection bias requires that we have a clear articulation of the theory we are testing. Is Kurzweil defending a guaranteed doubling of transistor density every 18 months indefinitely or is he defending something more general about compute power growing exponentially over time? Is the claim that compute power has been doubling every 18 months since the big bang and will do so until we enter a black hole or just that we are in a part of the production curve of computational technology that has this property? Without a fairly specific claim it is really hard to have any meaninful debate.
Is Moore's law a natural constant like the speed of light, an artifact of US capital markets in the 20th century, or an emergent clock signal for a complex interdependent system in which a lot of independent actors need to coordinate in order to avoid substantial pain. Are there things we can do to speed this rate up or to slow it down? Personally, I favor the clock signal explanation. If chip performance were higher than expected, the chip manufacturers would face substantial economic loss because of bottlenecks later in the supply chain to the end customers. If the performance were lower than expected, the economic pain would be felt in the supply chain after the chip manufacturers. The number of parties required to coordinate for chip production/consumption is sufficienbtly large that the heartbeat becomes self-reinforcing.
The emergent clock signal explanation looks a lot like Milton Friedman's explanation for inflation -- it is a reflection of the expectations of market participants about the value of money (and that Intel perhaps plays the same role in the chip supply that the Fed plays in money supply). It implies that the chip industry is vulnerable to the same sorts of disruptions that affect the money supply. If the government decided that a moonshot chip was required, the heartbeat would accelerate. If the government deregulated spectrum, the heartbeat might decelerate as capital was devoted to new bandwidth technologies rather than microcomputer CPUs. If some new discovery changed investor priorities (e.g. concern about luddite terrorists attacking chip fabs or chip investors) or consumer priorities (e.g. a desire for more passive entertainment, failure to build reliable power plants, ....) then new chip fabs might not be able to get sufficient capital to keep up their current growth rate. Or at an absurd level, if life on earth got wiped out by an asteroid, would Moore's law continue at the same rate? What is the claim that we are debating?