Keeping Pan responsive

Keeping an event driven program responsive when dealing with potentially long operations can be difficult.  There are two ways of doing this: multi-threading and asynchronous operations.  Each is best suited to different tasks with some overlap between them.  Network IO has already been handled mostly using async-io with threading being used for the connection setup due to api limitations.  File IO is a different story.  Pan's usage of files doesn't really lend itself to async-io therefore threading is used.  Well, at present only attachment saving is handled with other threads.  The other slow part is the loading & saving of the group header files.  Currently this code runs in the main thread which means that it stalls network io, especially when dealing with large groups which can easily take minutes.

A couple of articles by Herb Sutter have given me an idea for solving this problem, Active Objects and Prefer Futures and Callbacks.  The basic idea is to have an active object that handles loading and saving groups.  The hard part will be making the parts that need to load groups async.  The saving of groups might be easy if that is trgigered by a ref count.  The object would be something like this:

class GroupHandler {
  public:
    typedef std::tr1::function<void(group*)> Callback;
    void load_group(quark group, Callback cb)
      {queue.add(1,cb);}
    void save_group(quark group)

      {queue.add(2);}
    ~GroupHandler(){ join(thread); }

  private:
    virtual void do_load(quark group, Callback);
    virtual void do_save(quark group);

    bool done;
    Queue queue;
    Thread *thread;
}



strings & memory

In trying to improve the performance of Pan one of the places that I noticed a potential problem was in building newsrc lines.  The code was writing a read article range, '123-3214', into a buffer and then appending that into a std::string.  I used valgrind and kcachegrind to profile the code.  The problem with this is that each append would cause a new string to be allocated & then old one to be copied into it.  As the string becomes longer this takes more time & also causes the address space to be fragmented.

  My first attempt at optimizing this was to use a std::deque for temporary storage of the entire string then copy that into the output string as the final step.  Profiling showed varied results from slower to significantly faster.  The slow down was cause by the iterators used during the copy.  For short strings they dominated the performance of the function, however as the final string became larger the time savings of not having to allocate & copy more than made up for this.

  After realizing this i decided to profile my newsrc files to determine the line lengths I was using.  The results indicate that about 75% of the lines were under 256 bytes.  This was to small for the deque to be useful.  This time I decided to use a string for the temporary and reserve an initial 256 bytes so that most of the lines could be written without all the copying.  The profile showed a definite speedup doing this.  I could actually optimize this further by making the reserved space a function of the number of ranges in the line and a fixed estimate of the number of bytes per range.  This would nearly eliminate the need for a reallocation but i haven't tried this.

python, windows & unicode

Some time ago I wrote a backup program in python.  Recently I found a problem in that it wasn't handling non 8-bit characters in file names.  I finally tracked down part of the problem to not calling the walk function with a unicode path.  Of course I also needed to use the wide char versions of the windows api functions I was using to copy the files and create hard links.  I suppose I should see if this still applies with python 3.

Parsing XMP with python & rdflib

I have a python script that creates a web gallery. Currently it uses data from IPTC info in the files. I've been wanting to add XMP support to it for some time but couldn't find a solution. The adobe sdk is in C++ and doesn't work with python. The exempi project re-wrote it in C for use with python but it's apparently tied to pthreads on *nix. Since I don't need to create XMP just read it I started looking at RDF parsers for python and found rdflib. To get the XMP data from the images I wrote a script using info from the adobe sdk regarding how it is stored in various file types.

First off XMP is a subset of RDF encoded in XML. RDF deals with data in triples ( subject, predicate, value) which form a graph. Lists and arrays are formed by triples with the same subject and one of the predicates is 'type' with a value of 'Alt','Bag','List',or 'Seq'. the other predicates are either empty or are '_1','_2' etc. which designates the order of entries.

The first problem I ran into was a lack of documentation on rdflib. The only tutorials / examples I could find dealt with retreiving a single value which doesn't help when most of the data I needed was Alt or Bag. The second problem was that the parser was choking on the data I gave it which contained XML other that RDF. It just occured to me that I should have tried load instead of parse. Anyway, here is the method I've used to retreive data:

def getData(graph, pred):
 ''' '''
 try :  x=graph.objects(None, pred).next()
 except StopIteration : x=None
 if not isinstance(x,BNode) : return x
 ns=Namespace(RDF.RDFNS)
 val=graph.value(subject=x,predicate=ns['type'])
 if val in [RDF.Alt,RDF.Bag,RDF.Seq,RDF.List] :
     return Seq(graph,x)
 return None

Site update & Openid

After a very long time, I finally got around to updating my site. There shouldn't be much if any visible changes though. The update was mainly about using some css from YUI to ensure a consistent look across browsers and to use css for the nav bar roll-overs. Of course this did require a few js changes as well. The other change was to add limited OpenID delegation to my site. Now I can use it as my OpenID, at least with sites that support version 2.