subversion checksum failed

Well, it happens occasionally that in process of committing changes to the repository some weird problem with bad checksum is reported and the whole commit is blocked:


Commit failed (details below):
svn: Checksum mismatch for '/home/userdir/devel/RTTWeb-etree-branch/cherry/sql/.svn/text-base/Job.py.svn-base', expected: '947e06aeba5bd534250ad1124e3f60d6', actual:' a226478ea1900a7aebdb3bb0cb2d42d1'


How to fix it and don't lost your precious changes? Two ways:

1. VERY, VERY, VERRRY BAD:
   
   
   
   • go to the .svn subdirectory, in above example /home/userdir/devel/RTTWeb-etree-branch/cherry/sql/.svn/
     
   • add yourself write right to entries file and open it for editing
     
   • now very curious find and replace 'actual' checksum with 'expected' one, but don't touch any other line/word/character in that file
     
   • save & commit again
     
   
   
   
2. SAFE, CORRECT & ONLY WAY:
   
   
   
   • make a copy of working copy somewhere:
     cp -R /home/userdir/devel/RTTWeb-etree-branch/cherry/sql /home/userdir/backup
     
   • remove wrong file from working copy:
     rm /home/userdir/devel/RTTWeb-etree-branch/cherry/sql/Job.py
     
   • update working copy from server:
     svn up
     
   • copy back 'wrong checksum' file to working copy:
     cp //home/userdir/backup/Job.py /home/userdir/devel/RTTWeb-etree-branch/cherry/sql
     
   • commit:
     svn ci -m "recovered from checksum error"
     
   
   

Now guess which way to choose and why I'm so smart now? ;)

Compiz, whay not?

A few years ago my colleague had shown me Beryl window manager with all this super-duper spinning workspaces, 3D shaking transparent windows on rotating cubes. It sank deeply into my memory. I've tried several times to install it on MBP running Fedora, unfortunately w/o success. Up to now. This what I did:

installation

sudo yum install yum install mesa-dri-drivers-experimental xorg-x11-drv-nouveau \
compiz-fusion compiz-fusion-extras compiz-fusion-extras-gnome \
compizconfig-backend-gconf compizconfig-python \ 
emerald emerald-themes fusion-icon fusion-icon-qt \ 
libcompizconfig protobuf

reboot to load new graphic adapter drivers

switch compiz on from top-level gnome menu "System -> Preferences -> Desktop Effects"

let is auotstart at boot "System->Preferences->Startup Applications->New" and put there:

Name: whatever, Fusion Icon 
Application: fusion-icon --force

and save


Ta-dam! It's there. It's 5 minutes job, everything works out of the box.

Subversion server running on Fedora 13

Running subversion server using apache daemon should be an easy task, except you're security n00b like me.

1. Installing required binaries:
   sudo yum install subversion httpd mod_dav_svn
   



2. mod_dav_svn creates scratch configuration file subversion.conf for httpd daemon in
   /etc/httpd/conf.d/. Unfortunately in this file there is a bug in comments dealing with new repository creation:
   
   

   
   #
   # Example configuration to enable HTTP access for a directory
   # containing Subversion repositories, "/var/www/svn".  Each repository
   # must be both:
   #
   #   a) readable and writable by the 'apache' user, and
   #
   #   b) labelled with the 'http_sys_content_rw_t' context if using
   #   SELinux
   #
   
   #
   # To create a new repository "http://localhost/repos/stuff" using
   # this configuration, run as root:
   #
   #   # cd /var/www/svn
   #   # svnadmin create stuff   
   #   # chown -R apache.apache stuff
   #   # chcon -R -t http_sys_content_t stuff
   

   
   Last line should rather set SELinux context to http_sys_content_rw_t:
   

   
   chcon -R -t http_sys_content_rw_t stuff
   

   
   But anyway let's set our httpd configuration to be:
   
   
   
   • path for all repositories /var/www/svn
   
   
   • for apache authentication I choose the simplest one Basic with password file stored in /var/svn/passwd
   
   
   • finally I choose to store global svn authorization file in /var/svn/svnauth
   
   
   
   so using above presettings our subversion.conf would look like:
   

   
   LoadModule dav_svn_module     modules/mod_dav_svn.so
   LoadModule authz_svn_module   modules/mod_authz_svn.so
   <Location /repos>
      DAV svn
      SVNParentPath /var/www/svn
      AuthzSVNAccessFile /var/svn/svnauth 
      SSLRequireSSL
      Order deny,allow  
      AuthType Basic
      AuthName "Subversion repository"
      AuthUserFile /var/svn/passwd
      Require valid-user
   </Location>
   ## logger for svn: /var/log/httpd/svn_log
   CustomLog logs/svn_log "%t %u %{SVN-ACTION}e" env=SVN-ACTION
   

   
   



3. apache authentication file is of course produced using htpasswd command:
   

   
   # for 1st user we're creating a file (-c) and choose MD5 encryption (-m)
   sudo htpasswd -cm /var/svn/passwd cibak 
    New password: xxxxx
    Retype new password: xxxxx
    Adding password for user cibak
   # all the others users only added with the same encryption
   sudo htpasswd -m /var/svn/passwd jack
    ...
    ...
   

   
   Once this file is ready we have to set correct SELinux policy:
   

   
   sudo chcon -t httpd_sys_content_t /var/svn/passwd
   




4. creation of a new repository:
   

   
   sudo svnadmin create /var/www/svn/myrepo
   sudo chown -R apache:apache /var/www/svn/myrepo
   sudo chcon -R -t http_sys_content_rw_t /var/www/svn/myrepo
   

   
   



5. creation of subversion authorization file /var/svn/svnauth in what ever you choose editor (I prefer emacs):
   

   
   # cibak has read-write rights to the whole repository, jack could only read 
   [myrepo:/]
   cibak = rw
   jack = r
   # but jack is able to write in his own directory  
   [myrepo:/jack]
   jack = rw
   

   
   Syntax of this file is better described in the Subversion bible.
   
   Of course ones again we should remember to set correct SELinux policy context:
   

   
   sudo chcon -t httpd_sys_content_t /var/svn/svnauth
   

   
   



6. restarting of httpd deamon is a last step to switch our repo on:
   

   
   sudo /etc/init.d/./httpd restart
   

   
   

et voila, our repository is accessible under https://localhost/repos/myrepo URL.

Fedora 13 on MBP

Fedora 13. Download iso, put into CD drive, choose upgrade, wait for about 40 minutes, reboot, update GPT/MBRin reFit, boot. Whole procedure took me about one hour. Niiice!

But of course it could be harder if you want to install it from scratch. I had Fedora 12 installed already, so it was quick and easy task.

Welcome to Fedora 12!

I had tried to install Fedora 6 on my MacBook Pro 2-3 years ago but those days you have to hack to many things inside the kernel and config, that I gave up after one week of struggling. So, I have to use this %$#$%@ MacOSX, which looks like OS for emo teenager. I mean this young true emo kid, who is cutting himself with a pink razor to see his own artificial blood made of ketchup shouting and crying about Miss Spears private and public life. All this fancy jumping and zooming in and out icons and all this very deeply hidden options/configuration files...

Hopefully in the meantime good guys from Fedora were working on a new releases. And once Fedora 12 was released (well, I should say once I have realised it) I had gazed my eyes onto it eager to try it out once again. So I've downloaded x86_64 DVD ISO image, burned it and kept it at my fingertips, but the next step was o download and install in MacOSX small application called rEFIt. Afterwards I've downloaded and burned ISO image of gparted, moved, just for curiosity all my files to external USB hard drive and the repartitioning started straight away... I have shrunk main HD hpfs+ partition to 20 GB and created small /boot (300 MB), bigger /root (30GB) and rather huge /home (the rest) partitions, all using ext4 format. I've also created really small swap (2GB) at the very end of disk.

Then I put Fedora 12 installation DVD into slot and installation started. When it completed new lovely Tux has shown up under rEFIt menu just after resyc'ing HD master boot record with GUID. Then I logged to MacOSX once again only for changing boot options and choosing Fedora by default. I can't remember what I exactly did, but all details are in rEFIt documentation.

And then I've rebooted my laptop once again waiting with my fingers crossed to boot into Fedora 12 for the very first time.

What works out of the box? Built-in camera, track pad, ethernet and wireless cards, graphic adapter, special keys for LCD brightness, sound volume and keyboard backlight.

For the sound I have to tweak it a little putting into
/etc/modprobe.d/sound file this config:

alias snd-card-0 snd-hda-intel
alias sound-slot-0 snd-hda-intel
options snd-hda-intel model=mbp3

and reboot it once.

Then I set up Polish keyboard mapping (System → Preferences → Keyboard → Layout → Add and look for Polish, then modify Options for input of Polish characters, AFAIR I've ticked "Any ALT key" in "Key to choose 3rd level" option). Et voila: ąśźżłóę. :)

Then I went to standard procedure of installation of VERY IMPORTANT APPS (krb5, openafs, LDAP for Thunderbird etc.) googling for their configuration in cern.ch domain.

And that's it! Since one month I'm working on Fedora 12. And I'm very happy!

Post-installation took me some more time, especially that Fedora guys are really good guys and they're not distributing any piece of code which has got some problem with the Law (so i.e. there are no mp3 codecs in official Fedora yum repos etc.). Fortunately all is quite good documented on Internet by not-so-good guys and easy to find thanks to Google, Inc.

Farewell MacOSX! Never see you again! Never ever come back!!!

CherryPy behind blue eyes of apache

Well, a big change is coming. We are going to switch our web app from PHP+jQuery to CherryPy+jQuery+some kind of ORM+some db back-end (and with a little intrusion of PHP, but #!).

It won't be easy... First of all I've put all parts of a new system together:

• CherryPy 3.1.2 for "pythonization"
  
• SQLObject 0.10.2 as out ORM
  
• Cheetah 2.2.2 for templates
  
• sqlite or mysql for db back-end (still not decided)
  

I want to run CherryPy application in a standalone server, probably as a deamon, on our www host and communicate with it using apache + mod_proxy.

1. deamonization
   
   Sound easy but it took me two days to understand cherryd usage. After that I've wrote a shell script to run my CherryPy application as a deamon under Fedora (I hope it would work also under RHEL..):
   
   

   
   #!/bin/sh
   #
   # Comments to support chkconfig on RedHat Linux
   # chkconfig: 35 96 96
   # description: Cherrypy start/stop script
   #
   #
   ### BEGIN INIT INFO
   # Provides: cherrypy 
   # Required-Start: $local_fs $syslog
   # Required-Stop: $local_fs $syslog
   # Default-Start: 2345
   # Default-Stop: 90
   # Short-Description: run CherryPy web server
   # Description: CherryPy is a python framerwork for CMT 
   #
   ### END INIT INFO
   
   . /etc/init.d/functions
   
   serverdir=/home/cibak/CheRTTyPy
   prog=get
   server=${serverdir}/get.py
   config=${serverdir}/conf/get.conf
   pidfile=${PIDFILE-/var/run/CheRTTyPy.pid}
   lockfile=${LOCKFILE-/var/lock/subsys/CheRTTy}
   
   PYTHONPATH=${serverdir}
   
   RETVAL=0
   
   start() {
       echo -n "Starting $prog:"
       cherryd -d -p ${pidfile} -c ${config} -i get
       RETVAL=$?
       [ $RETVAL = 0 ] && touch ${lockfile}  
       echo_success
       echo
       return $RETVAL
   }
   
   stop() {
       echo -n "Stopping $prog:"
       if test -f ${pidfile}; then
           killproc -p ${pidfile} -d 10 $prog
           rm -f "$pidfile"
           RETVAL=$?
       else
           RETVAL=1
       fi
       RETVAL=$?
       echo_success
       echo
       [ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile}
   }
   
   
   case "$1" in
        start)
           start
           ;;
        stop)
           stop
           ;;
        status)
           status -p ${pidfile} $server
            RETVAL=$?
           ;;
        restart)
    stop
    start
    ;;
        *)
           echo $"Usage: $0 {start|stop|restart|status}"
           exit 1
   esac
   exit $RETVAL
   

   
   
   
2. apache and mod_proxy
   
   I want to run CherryPy app behind apache and use mod_proxy to redirect ajax calls to it. So the vhost configuration:
   
   

   
   <VirtualHost cibak.fedora12.mbp:443>
       ServerAdmin krzysztof.ciba@xxxxxxxxx.xxx
       ServerName cibak.fedora12.mbp
       ProxyPreserveHost On
       
       DocumentRoot /home/cibak/CheRTTyPy
       ErrorLog logs/cibak.fedora12.mbp_error.log
       TransferLog logs/cibak.fedora12.mbp_access.log 
   
       SSLEngine On
       SSLProtocol all -SSLv2
       SSLProxyEngine On
   
       SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
   
       SSLCertificateFile /etc/pki/tls/certs/localhost.crt
       SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
      
       <Proxy *>
           Order allow,deny
           Allow from all
       </Proxy>
       ProxyPass /query/ https://127.0.0.1:8083/
       ProxyPassReverse /query/ https://127.0.0.1:8083/
   </VirtualHost>
   

   
   
   
3. ORM
   SQLObject, looks pretty nice, especially sqlbuilder classes for constructing SQL query. For prototype I choose simple sqlite3 DB-API, as it is already built in python, but for production version I'll switch to ORACLE or mysql.
   
   BTW switch of db back-end is very easy with SQLObject ORM — only place for that is changing connection string passed to sqlconnector. :)
   

After 6-7 weeks of rather extensive work the prototype for Atlas experiment RTT web page is ready and released to the public. Now I'm waiting for the feedback.



!!!WORK IN PROGRESS!!!

Personas for Firefox

Want to be a better Atlas member? Want to be a cool cat in your office? Want to impress your friends around? Use
this! ;)

reallocate your code from one repository to another

I'm a bit too much python fan, so once again it would be a python script. Sorry! ;)

Imagine you want to move your code from one Subversion repository to another with all history preserved, but under different directory. No problem at all.

1. First you need to dump contents of, let's say, source repository to the dump file, probably filtering out someone else's stuff:
   
   

   
   $> svnadmin dump /path/to/src/repo | \
   svndumpfilter --drop-empty-revs --renumber-revs \ 
   --skip-missing-merge-sources include SRCDIR > SRCDIR.dump
   




2. Now filtering script:
   
   

   
   ##
   # @file realloc.py  
   # @author Krzysztof Daniel Ciba (Krzysztof.Ciba@NOSPAMgmail.com) 
   #
   import sys, os
   import optparse
   ##
   # @class realloc
   # @author Krzysztof Daniel Ciba (Krzysztof.Ciba@NOSPAMgmail.com)
   # @brief reads stdin dump file, changes all paths from SRC to DEST and prints it out to stdout
   class realloc( object ):
   
       ## c'tor
       # @param self "Me, myself and Irene"
       # @param src source path in dump file
       # @param dest destination path on dump file
       def __init__( self, src,  dest ):
           self.src = src
           self.dest = dest
           
       def run( self ):
           for line in sys.stdin:
               line = line.strip("\n")
               if ( "Node-path:" in line or
                    "Node-copyfrom-path:" in line ):
                   if self.src in line:
                       line = line.replace( self.src, self.dest )
               print line
               
   def check( option, opt_str, value, parser ):
       if ( str(value).startswith("/") ):
            raise optparse.OptionValueError( "value of " + opt_str + " should be a relative path!" )
       setattr( parser.values, option.dest, value )
   
   ## trigger processing
   if __name__ == "__main__":
       version = "%prog $Revision: 8715 $ by Krzysztof Daniel Ciba (Krzysztof.Ciba@NOSPAMgmail.com)"
       usage = "%prog [opts] [< indumpfile > outdumpfile]"
       parser = optparse.OptionParser( usage=usage, version=version )
       parser.add_option("--src", type="string", action="callback", callback=check, dest="src", help="source directory" )
       parser.add_option("--dest", type="string", action="callback", callback=check, dest="dest", help="destination directory" )
       opts, args = parser.parse_args(sys.argv[1:])
       if ( not opts.src and opts.dest ):
           parser.error("option --dest present but --src is missing")
       elif ( opts.src and not opts.dest ):
           parser.error("option --src present but --dest is missing")
       elif ( not opts.dest and not opts.dest ):
           parser.error( "options --src and --dest are required" )
       else:
           theApp = realloc( opts.src, opts.dest )
           sys.exit( theApp.run() )
   




3. and it's action
   
   

   
   $> realloc.py --src SRCPATH --dest DESTPATH < SRCPATH.dump > DESTPATH.dump
   




4. and standard loading to new repository:
   
   

   
   $> svnadmin load /path/to/new/repo < DESTPATH.dump
   

Not so much work, especially it could be used in pipes, i.e. dumping, filtering, reallocating and loading in the same time:

$> svnadmin dump /path/to/src/repo | svndumpfilter \ 
--drop-empty-revs --renumber-revs --skip-missing-merge-sources \ 
include SRCDIR | realloc.py --src SRCDIR --dest DESTDIR | \ 
svnadmin load /path/to/dest/repo  

Hmmm... Of course it could be made better in some simple way using sed. Yes, I'm using sed from time to time, but NOT this time. ;)

hacking cvs2svn ;)

Several months ago my BOSS asked me to transfer our ancient CVS repository to the brand new Subversion system. Not a big deal -I thought - there is lovely tool which does it for me: cvs2svn converter made by Tigris. But just after that I realized it wouldn't be such easy task, as our current repository is rather huge and holds several years of our projects history. Some fraction of packages was obsolete and additionally there was a request to split it to three different SVN repos and keep in a newly Subversion repos only recent year of history.

Not a big deal once again - I said after reading of cvs2svn documentation - we've got a database holding all the cvs tags used to built our software, so I'd make a few queries to know what to keep and that's all, except... OMG! There is no "include" pattern for tags!!!

You could in a easy way exclude symbols from cvs, which shouldn't be converted, but I've got completely opposite situation. Now, smart guys, please try to write me a reqexp which is nagative to a pattern... Oups!

What I did it was a little modification of cvs2svn tool in cvs2svn/cvs2svn_lib/symbol_strategy.py:

Then of course I wrote a script which:

• asked our tag db for packages and their tags/branches that should be migrated
  
• generated cvs2svn option file
  
• and run cvs2svn tool
  

Several testing later a big migration day arrived on agenda. I've started to migrate about 9:00 AM and before 6:00 PM ca. 2k packages were moved generating initially about 190k revisions. And one day later we've opened new repos to the whole collaboration.

SVN hooks once again, this time tags, branches, log messages etc.

This one hook I wrote quite long ago is forcing users of Subversion repository to:

• provide log message on every commit (class logChecker)
  
• block removing of "/tags", "/branches" and "/trunk" directories from repo (class pathChecker)
  
• forcing branch and tag naming convention
  
• making subdirectories under /tags read-only
  

My company is running Subversion for storing all software packages and our repo hasn't got the default structure (/tags, /trunk and /branches as a top directories) — instead of that every package has got it's own /trunk, /tags and /branches subdirectories. So the repository structure is something like this:

/Path/
      PackageA/
               trunk/ 
                     src/
                     doc/
               tags/ 
                    PackageA-00-00-00/ 
                                      src/
                                      doc/
                    ...
               branches/
                    PackageA-00-00-00-branch/ 
                                            src/
                                            doc/
                  
      PackageB/
               ...
/OtherPath/
           PackageC/
           ...                   

Looks rather complicated, but it isn't. ;)

Also we've got tags and branches naming convention. Every tag should be made of package name, then a three (or four for tags made over a branch) groups of digits, e.g.: PackageName-ii-jj-kk or PackageName-ii-jj-kk-ll.
Every branch name is very similar to tag name, except there is additional (-branch) string at the end.

So here is the hook itself:


If you want to use this one, you have to modify it a little:

• replace 'librarian' or/and 'root' account name with your own repository admin
  
• put repository admin mail to block messages - replace this fake 'librarian@mail' address
  
• change regexp ffor tags and branches to match your criteria
  
• put svn-policy.py script into your repository hook directory ($REPO/hooks)
  
• modify "$REPO/hooks/pre-commit" script to switch this one on
  

  
  #!/bin/sh
  REPOS="$1"
  TXN="$2"
  SVNPOLICY=/path/to/repo/hooks/svn-policy.py # modify this line
  #SVN policy
  $SVNPOLICY  -t "$TXN" "$REPOS" || exit 1
  # All checks passed, so allow the commit.
  exit 0
  

  
  

Maybe the code needs some cleaning and better structure but anyway happy using, comments and questions are welcome!

Cheers,
Krzysztof

rendering graphs in PHP using GraphViz

Well known and very popular graph rendering software Graphviz has got many different language bindings and is widely used in many different areas of maths and computing. I don't have to speak more. It just renders all those graphs in easy way using very simple but powerful language.

Amongst various tools which are using Graphviz there is also a PHP package in PEAR — Image_GraphViz, which has got only one pitfall — it cannot render cluster (a subgraph) inside a cluster (at least not in 1.2.1 version). So, I modified it a bit to make it possible:



Here is the results of running above example on apache (it renders image in svg format, so not all web browsers are able to display it correctly):



Test Graph in SVG

I was very happy when I'd figured out that my changes went into the official Image_GraphViz 1.3.0RC3. Good luck folks!