Overview over all blog items

01.02.2007

ZSQLAlchemy

A proof-of-concept integration of SQLAlchemy into Zope 2

Over the last few days I played around with SQLAlchemy in order to find ways out of the SQL hell. As a result (or call it a quick hack)  I implemented a  wrapper  ZSQLAlchemy that integrates SQLAlchemy with Zope 2. The wrapper (right now) only cares about the connection and transaction handling and exposes a session and a metadata object (which is usually enough for writing applications based on SQLAlchemy). The session participates in the transaction handling of Zope. This means that a Zope commit will also trigger a session.flush() operation.

ZSQLAlchemy will never be a higher-level framework (e.g. like the (over-engineered) Alchemist framework for Plone).  You can compare it with a standard Zope database adapter.

You might grab ZSQLAlchemy from svn.zope.org and play with it. At the moment it will work with Postgres only. Support for other databases (the ones supported by SQLAlchemy) can be added easily.


12.11.2005

Zope on Windows: Fun, Fun, Fun

A nightmare introduction running external programs on Windows

Introduction

For a Zope recent project I was in charge to develop a solution to generate PDF and RTF from HTML. I choose XSL-FO as general framework to provide a homogeneous environment for such transformations and conversions. There are bunch of FO processors on the marked and I took some commercial Java-based FO processors. Early tests worked fine and the first implementations worked fine from within Zope on Linux. The content presented to the customer could be transformed on the-fly to plain text, PDF or RTF. But fun started when I was trying to get the stuff working on Zope unter Windows. The target platform for the whole application is Windows (from Windows XP down to Windows '98).


Let's get technical

Running external processes from Python is usually not a big deal. You have different options:

os.system()

command.getoutput()

os.popen()

subprocess.Popen()

...and some more variations. Basically all methods provide a way to run a command like

java -jar some.jar arguments

This also works fine on XP but the fun starts when you want to run the same code under different Windows flavours. My very first implementation was like this:

P = subprocess.Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE; stdin=None)

status = P.wait()

stdout = p.stdout.read()

stderr = p.stdout.read()

This works fine on XP but not on Windows 98.  The issue that came up was an exception from the subprocess module where Python tried to obtain a handle for stdin internally through some win32 API call. So the first workaround was:

P = subprocess.Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE; stdin=open('nul:')

Using this workaround I was able to run java.exe -version to check for the installed Java version. However using this code caused an empty DOS window to popup for every execution. The solution was to use the startupinfo parameter:

startupinfo = STARTUPINFO()

startupinfo.dwFlag |= STARTF_USESHOWWINDOW

P = subprocess.Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE; stdin=open('nul:'), startupinfo=startupinfo)

Now with this solution I could execute Java command without anyone noticing it :-) But....wait....our Java commands also produced some output..maybe only some lines or a hundred warnings..you would expect that you can read the error messages from the stdout and stderr pipes....surprise, surprise....that worked only for short messages. In case the Java process wrote too much data to stderr then whole process was just hanging with the result that os.wait() had to wait forever. Because of my limited Windows knowledge I could not track down this problem on Windows 98. So the workaround at this time was to avoid using stderr and stderr as PIPEs and just setting them to None.  So with the final solution I was able to run all our converters using the same code on all Windows flavours...but I really took me very long (serveral days with lots of iterations with external testing labs) to figure this all out....and it convinced me again that Windows just a piece of crap...



Plone still has a poor support for storing large files. We all know that the ZODB badly supports the storage of large file (BLOB support within the ZODB is coming forward but it is not ready for prime time). For recent project we had the requirement to build a multimedia database as part of a Plone application. The database should be able to store several 100K  objects (images, media files) inside Plone. We choosed FileSystemStorage from Ingeniweb because it fits perfectly into the storage layer concept of Archetypes. Creating new types (derived from ATFile or ATImage) using the filesystem for storing large content could be implemented very easily.

Adding FSS support to your own content-type is very easy:

def updateSchema(schema):
field = schema['file']
field.storage = DirectoryFileSystemStorage()
field.registerLayer('storage', field.storage)

class MyFile(ATFile):

schema = ATFile.schema
updateSchema(schema)
.....

FSS by default supports only one Plone site per Zope instance since it stores the data as flat hierarchy directly in the var folder. We extended FSS in the following ways:

  • support for multiple Plone sites within one Zope instance
  • two-level deep directory structure based on the objects UIDs
  • configurable storage prefixes
  • working copy/cut/paste support
  • generated RDF contains more metadata (review state, effective, expires dates)

Most of the work was done by Simon Pamies (Applause).

The code is not fully polished. Since we were only interested in the FSS backend functionality we don't made any efforts in order to work on the Plone UI of FSS. There is a big chance that this functionality is broken. However the backend code is working fine even for some 100K objects.

A snapshot of the code is available from ftp://ftp.zopyx.com/open-source/FSS.tar.gz


30.07.2006

Kupu must die

Plone ships with Kupu since ages. However I think Kupu really sucks and there are better alternatives. We tried to make our last customer happy with Kupu they really did not like it. We've been looking at alternatives and come over FCKEditor. It has also a seamless Plone integration that works without problems. Just download the archive, install it and replace Kupu in your personal member preferences with FCKEditor. It's really much more powerful than Kupu. Plone should really ship with FCKeditor in future versions. Kupu is just annoying.


I have been working on a new TextIndexNG3 release today that will basically contain two fixes:

  1. The phrase search could possibly return fault positives. The phrase search was implemented as a substring search on the list of word ids that is encoded using the WidCode.py module of ZCTextIndex. Under certain circumstances an encoded substring could be found although I did not represent a valid search result.
  2. For large documents the encoded list of word ids night become long. In all older versions these strings have been stored as values of an OOBTree. This might cause large transactions since the ZODB organizes its data in buckets. Imagine you have a bucket with 20 strings where each string has 1 MB of data. When you modify a single string the complete bucket (roughtly 20MB) would have been transfered to the ZODB. As a solution all strings are now wrapped as a persistent subobject that can be loaded and stored individually.
Both errors were reported by Dieter Maurer and affect the textindexng.storage.Storage class implementation. Thanks!

TextIndexNG 3.1.13 is supposed to be released this weekend or early next week.

I've spent some time over the weekend to clean up my zoo of Plone instances for my ZOPYX domains. All sites were migrated into a single Zope 2.9.4 instance running Plone 2.5.

Visible changes are:

  • new site opensource.zopyx.com: this site runs PloneHelpCenter and PloneSoftwareCenter and will replace the existing opensource.zopyx.biz in the near future.  With this migration I will retire my own SoftwarePackageNG package because PloneHelpCenter and PloneSoftwareCenter became what SoftwarePackageNG was supposed to be.
  • blog.zopyx.biz moved to blog.zopyx.com and now uses Quills 1.5. This site is also open now for joining in case you want to comment on a blog entry.
  • www.zopyx.com remained unchanged
To be honest: the migration went very smoothly compared to any other Plone migration in the past. I only had to tweak the i18n machinery
at one point because of some UTF-8 unicode conversion errors..not a big deal.

    14.08.2006

    Dynamic Archetypes forms

    This blog entry gives a short introduction about making static Archetypes forms more dynamic

    Autogenerated Archetypes forms (generated through the base_edit view) are in general boring static. Often you have the use-case to make forms more dynamic. What means dynamic? Dependent on the value of a particular form element you might be interested e.g. to show or hide other elements of the form. In a recent project we had the case where the user could choose a particular publication mode (daily, weekly, monthly, bi-monthly) for a given document. The widgets for the different publication modes were somewhat complex and bit large since a single widget did consist of serveral other checkboxes. select elements and input fields. It was the customer's wish to present only only one publication widget  that matched the value of the particular publication mode.

    So how to approach this problem?

    The solution is pretty easy: Javascript + CSS. Using the prototype.js Javascript library (a good documentation can be found here) and the Plone Javascript resource registry it was easy to add an event handler to the select element that controls the publication mode:

      Event.observe(select_id, 'click', observer);
    This tells the browser to call the method observer() for every change on the element with the ID select_id. The implementation of the observer() method basically looks like this:

    function observer() {
    var v = $F(select_id);

    if (v=='weekly')
    show_weekly_control()

    if (v=='monthly')
    show_monthly_control()
    ....
    }

    So everytime you cange the value of the select element the actual value of select element ($F() is a shortcut method (provided by prototype.js)  to obtain the value of a form element independent of its type) is stored in the variable v. For every possible value of v there is a dedicated method that controls the visibility of other form elements:

    function show_weekly_control() {
        show_element('archetypes-fieldname-control_weekly');
        hide_element('archetypes-fieldname-control_monthly');
    }

    function show_monthly_control() {
        hide_element('archetypes-fieldname-control_weekly');
        show_element('archetypes-fieldname-control_monthly');
    }

    function show_element(id) {
    if ($(id))
    $(id).style.display = 'block';
    }

    function hide_element(id) {
    if ($(id))
    $(id).style.display = 'none';
    }

    The archetype-fieldname-XXXXX IDs are autogenerated IDs (by Archetypes) for every field of a schema.

    Using this approach we are able to hide and show form elements for general Archetypes forms without hacking any Archetypes or Plone code. I know that this not a complete how-to but it shall point you in the right direction to make more out of your boring Archetype forms.

    Although I haven't had any plans to continue the development of PloneCollectorNG in the near future, I found some time to merge a huge patch provided by Jean-Charles Rogez back into the V 1.2 code base. The result is now available as a beta release for  V 1.2.10. This version should work with Plone 2.1 and 2.5 (although the focus of my work was on making this release compatible with Plone 2.5). The beta release and the complete release notes can be found at:



    Apple  shipped over a million crappy batteries produced by Sony for their iBooks and Powerbooks. Now they are following the road of DELL  and they  are running a battery exchange program.

    Fortunately or unfortunately I am elegible for the exchange program. So my Powerbook is about 18 month old and I am already planning to buy a Macbook Pro soon. The running time of the Powerbook with both batteries was down to 60-90 minutes (compared to 4-5 hours when they were new) which is typical for batteries of that age. Now getting two new batteries is really a perfect service and increases the resell-value of the Powerbook. Apple rocks!!!

    But remember to unplug your power chord while leaving your Powerbook unattended :-)