strings & memory

In trying to improve the performance of Pan one of the places that I noticed a potential problem was in building newsrc lines.  The code was writing a read article range, '123-3214', into a buffer and then appending that into a std::string.  I used valgrind and kcachegrind to profile the code.  The problem with this is that each append would cause a new string to be allocated & then old one to be copied into it.  As the string becomes longer this takes more time & also causes the address space to be fragmented.

  My first attempt at optimizing this was to use a std::deque for temporary storage of the entire string then copy that into the output string as the final step.  Profiling showed varied results from slower to significantly faster.  The slow down was cause by the iterators used during the copy.  For short strings they dominated the performance of the function, however as the final string became larger the time savings of not having to allocate & copy more than made up for this.

  After realizing this i decided to profile my newsrc files to determine the line lengths I was using.  The results indicate that about 75% of the lines were under 256 bytes.  This was to small for the deque to be useful.  This time I decided to use a string for the temporary and reserve an initial 256 bytes so that most of the lines could be written without all the copying.  The profile showed a definite speedup doing this.  I could actually optimize this further by making the reserved space a function of the number of ranges in the line and a fixed estimate of the number of bytes per range.  This would nearly eliminate the need for a reallocation but i haven't tried this.

python, windows & unicode

Some time ago I wrote a backup program in python.  Recently I found a problem in that it wasn't handling non 8-bit characters in file names.  I finally tracked down part of the problem to not calling the walk function with a unicode path.  Of course I also needed to use the wide char versions of the windows api functions I was using to copy the files and create hard links.  I suppose I should see if this still applies with python 3.

Parsing XMP with python & rdflib

I have a python script that creates a web gallery. Currently it uses data from IPTC info in the files. I've been wanting to add XMP support to it for some time but couldn't find a solution. The adobe sdk is in C++ and doesn't work with python. The exempi project re-wrote it in C for use with python but it's apparently tied to pthreads on *nix. Since I don't need to create XMP just read it I started looking at RDF parsers for python and found rdflib. To get the XMP data from the images I wrote a script using info from the adobe sdk regarding how it is stored in various file types.

First off XMP is a subset of RDF encoded in XML. RDF deals with data in triples ( subject, predicate, value) which form a graph. Lists and arrays are formed by triples with the same subject and one of the predicates is 'type' with a value of 'Alt','Bag','List',or 'Seq'. the other predicates are either empty or are '_1','_2' etc. which designates the order of entries.

The first problem I ran into was a lack of documentation on rdflib. The only tutorials / examples I could find dealt with retreiving a single value which doesn't help when most of the data I needed was Alt or Bag. The second problem was that the parser was choking on the data I gave it which contained XML other that RDF. It just occured to me that I should have tried load instead of parse. Anyway, here is the method I've used to retreive data:

def getData(graph, pred):
 ''' '''
 try :  x=graph.objects(None, pred).next()
 except StopIteration : x=None
 if not isinstance(x,BNode) : return x
 ns=Namespace(RDF.RDFNS)
 val=graph.value(subject=x,predicate=ns['type'])
 if val in [RDF.Alt,RDF.Bag,RDF.Seq,RDF.List] :
     return Seq(graph,x)
 return None

Site update & Openid

After a very long time, I finally got around to updating my site. There shouldn't be much if any visible changes though. The update was mainly about using some css from YUI to ensure a consistent look across browsers and to use css for the nav bar roll-overs. Of course this did require a few js changes as well. The other change was to add limited OpenID delegation to my site. Now I can use it as my OpenID, at least with sites that support version 2.

Some musings on graphics tech

Good idea, bad timing.
I just read a post on dailytech about a new chip that's supposed to allow multi-gpu multi-vendor graphics with near linear performance scalling. The basic idea is to render different scene elements on each gpu and combine them. Draw a wall here a character there and so on. One of the problems mentioned in the article is that vista does not allow more than one graphics driver so you couldn't mix ati and nvidia cards even though the tech supports it. Another issue mentioned in the comments was that it's directx only, at least for now. What I'm wondering about is what happens when games start using there own renderers instead of the standard directx and opengl renderers? I'm pretty sure this won't work with such software.

On a related note what with all the opengl bashing? Seems a nmber of people want it to go away which doesn't make any since it's the only cross platform api that exists. It's used on all desktop OS platforms and most smaller devices including cell phones. Even Google's android will use it.