Archive 19. January 2005

Post 4000

Wow. This here is - in the current database - the 4000th article

erstauntes Gesicht

bigempty.com is a very nice photo blog with a very minimalist design. A nice gimmick: the background color of the footer area is based on the displayed photo - I have no idea if it's calculated automatically, but the idea is nice.

Federal Border Guard will soon be called Federal Police - wasn't there something about a ban on a federal police force for Germany? Anyway, this last line of shame is also falling.

Photos of torturing soldiers in Iraq shock the British - but will they tell their Prime Minister what they think because of it? And more importantly: will Blair finally distance himself from this madness? Or at least learn from it for the future and not participate in the Iran madness that is slowly emerging among the Americans?

Got New Spam Tactic Figured

Asymptomatic » Got New Spam Tactic Figured reports on a new tactic used by blog spammers. Relatively harmless comments appear on blogs that don't contain a single link. When spammers find these comments again via Google, they know they can likely post further comments there—bypassing the filters that automatically approve comments from visitors who have previously had a comment approved under their email address. So it could be that after a "Hey, I think your site is great" comment, a flood of blog spam suddenly appears...

Apparently stricter penalties for tailgaters planned - I think that's urgently needed too. I get upset every time we're on the motorway. It's crazy what some people allow themselves behind the wheel... (and no, we're neither chronic left-lane drivers nor slow crawlers)

RSS 1.1: RDF Site Summary (DRAFT)

RSS 1.1: RDF Site Summary (DRAFT) - how sensible. Someone designed an update to RSS 1.0 (yes, that unloved RDF-based format that hardly anyone really knows). Because, there aren't enough feed formats yet, so you definitely have to add one more.

Along those lines, I also stumbled upon the HTML Syndication Format - another thing like that, where someone thought it would be a good idea to deliver the feed as specially tagged XHTML source.

And unlike RSS 3.0 (which is based on YAML instead of XML), the other two seem to be more or less serious (with the HTML Syndication Format I'm still hoping it was just an elaborate April Fools' joke ...)

Schwarzenegger rejects clemency petition

Der Terminator macht seinen Filmfiguren alle Ehre - and sends another person to their death.

Secure and anonymous on the Internet with proxies - Guide to using Privoxy and tor. I hadn't linked it here somehow. It's so good, you should read it. You won't become any dumber from it.

Subtraction: New, Improved Original Flavor!

Khoi Vinh presents his new design. It's somehow cool how he creates — for me — quite an interesting impression through omission. I really like it anyway.

I wish I could come up with my own design and didn't always have to steal my designs from somewhere ... (one of the real advantages of WordPress - the Kubrick design may be quite widespread by now, but at least it can be adapted even by design dummies like me without everything falling apart)

WordPress NoFollow Plugin

The WordPress NoFollow Plugin adds rel="nofollow" to links in comments to remove their Google ranking. While I personally find it a shame that links in comments are generally not followed, thus removing the useful opportunity for smaller blogs to promote their own through active discussion in other blogs. Okay, in the end it's not that bad, but somehow a small piece of the "one link washes the other" mentality of blogs is lost... A small handicap is that the author has directly linked the plugin and unfortunately his server executes the PHP directly. At the moment, you can't download it, you only get an empty HTML page.

Zope Hosting and Performance - English Version

Somebody asked for an english translation of my article on Zope Hosting and Performance. Here it is - ok, it's not so much a direct translation than a rewrite of the story in english. Enjoy.

Recently the Schockwellenreiter had problems with his blog server. He is using Zope with Plone and CoreBlog. Since I am doing professional Zope hosting for some years now, running systems that range in the 2000-3000 hits per minute scale, I thought I put together some of the stuff I learnt (sometimes the hard way) about Zope and performance.

  • The most important step I would take: slim down your application. Throw out everything you might have in the Zope database that doesn't need to stay there. If it doesn't need content management, store it in folders that are served by Apache. Use mod_rewrite to seemlessly integrate it into your site so that people from the outside won't notice a difference. This can be best done for layout images, stylesheets etc. - Apache is much faster in delivering those.
  • Use Zope caching if possible at any rate. The main parameter you need to check: do you have enough RAM. Zope will grow when using caching (especially the RAMCacheManager). The automatic cleanup won't rescue you - Zope will still grow. Set up some process monitoring that automatically kills and restarts Zope processes that grow above an upper bound to prevent paging due to too large memory consumption. This is even a good idea if you don't use caching at all.
  • There are two noteable cache managers: one uses RAM and the other uses an HTTP accelerator. The RAMCacheManager caches results of objects in memory and so can be used to cache small objects that take much time or much resources to construct. The HTTPCacheManager is for using a HTTP accelerator - most likely people will use Squid, but you can use an appropriately configured Apache, too. The cache manager will provide the right Expires and Cache-Control headers so that most traffic can be delivered our of the HTTP accelerators instead of Zope.
  • Large Zope objects kill Zopes performance. When using caching they destroy caching efficiency by polluting the cache with large blobs of stuff that isn't often required and Zope itself will get a drain in performance by them, too. The reason is that Zope output is constructed in-memory. Constructing large objects in memory takes much resources due to the security layers and architectural layers in Zope. Better to create them with cronjobs or other means outside the Zope server and deliver them directly with Apache. Apache is much faster. A typical situation is when users create PDF documents in Zope instead of creating them outside. Bad idea.
  • Use ZEO. ZEO rocks. Really. In essence it's just the ZODB with a small communication layer on top. This layer is used in Zope instances instead of using the ZODB directly. That way you can run several process groups on your machine, all connecting to the same database. This helps with the above mentioned process restarting: when one is down, the other does the work. Use mod_backhand in Apache to distribute the load between the process groups or use other load balancing tools. ZEO makes regular database packs easier, too: they run on the server and not in the Zope instances - they actually don't notice much of the running pack.
  • If you have, use a SMP machine. Or buy one. Really - that helps. You need to run ZEO and multiple Zope instances, though - otherwise the global interpreter lock of Python will hit you over the head and Zope will just use one of the two processors. That's one reason why you want multiple process groups in the first place - distribution of load on the machine itself, making use of multiple processors.
  • You can gain performance by reducing the architectural layers your code goes through. Python scripts are faster than DTML. Zope products are faster than Python scripts. Remove complex code from your server and move it into products or other outside places. This needs rewriting of application code, so it isn't allways an option to do - but if you do, it will pay back.
  • Don't let your ZODB file grow too large. The ZODB only appends on write access - so the file grows. It grows quite large, if you don't pack regularily. If you don't pack and you have multi-GB ZODB files, don't complain about slow server starts ...
  • If you have complex code in your Zope application, it might be worthwile to put them into some outside server and connect to Zope with some RPC means to trigger execution. I use my |TooFPy| for stuff like this - just pull out code, build a tool and hook it into the Zope application via XMLRPC. Yes, XMLRPC can be quite fast - for example pyXMLRPC is a C-written version that is very fast. Moving code outside Zope helps because this code can't block one of the statically allocated listeners to calculate stuff. Just upping the number of listener threads doesn't pay as you would expect: due to the global interpreter lock still only one thread will run at a time and if your code uses C extensions, it might even block all other threads while using it.
  • If you use PostgreSQL, use PsycoPG as the database driver. PsycoPG uses session pooling and is very fast when your system get's lots of hits. Other drivers often block Zope due to limitations like only one query at a time and other such nonsense. Many admins had to learn the hard way that 16 listener threads aren't really 16 available slots if SQL drivers come into play ...

There are more ways to help performance, but the above are doable with relatively small work and are mostly dependend on wether you have enough memory and maybe a SMP machine. Memory is important - the more the better. If you can put memory into your machine, do so. There is no such thing as too-much-memory (as long as your OS supports the amount of memory, of course).

What to do if even those tips above don't work? Yes, I was in that situation. If you come into such a situation, there is only one - rather brutish - solution: active caching. By that I mean pulling stuff from the Zope server with cronjobs or other means and storing it in Apache folders and using mod rewrite to only deliver static content to users. mod rewrite is your friend. In essence you just take those pages that kill you currently and make them pseudo-static - they are only updated once in a while but the hits won't reach Zope at all.

Another step, of course, is more hardware. If you use ZEO it's no problem to put a farm of Zope servers before your ZEO machine (we currently have 5 dual-processor machines running the Zope instances and two rather big, fat, ugly servers in the background for databases, frontend with two Apache servers that look allmost like dwarves in comparisons to the backend stuff).

Zope is fantastic software - don't mistake me there. I like it. Especially the fact that it is an integrated development environment for web applications and content management is very nice. And the easy integration of external data sources is nice, too. But Zope is a resource hog - that's out of discussin. There's no such thing as a free lunch.