Strony

środa, 30 czerwca 2010

Subversion server running on Fedora 13

Running subversion server using apache daemon should be an easy task, except you're security n00b like me.


  1. Installing required binaries:
    sudo yum install subversion httpd mod_dav_svn


  2. mod_dav_svn creates scratch configuration file subversion.conf for httpd daemon in
    /etc/httpd/conf.d/. Unfortunately in this file there is a bug in comments dealing with new repository creation:


    #
    # Example configuration to enable HTTP access for a directory
    # containing Subversion repositories, "/var/www/svn". Each repository
    # must be both:
    #
    # a) readable and writable by the 'apache' user, and
    #
    # b) labelled with the 'http_sys_content_rw_t' context if using
    # SELinux
    #

    #
    # To create a new repository "http://localhost/repos/stuff" using
    # this configuration, run as root:
    #
    # # cd /var/www/svn
    # # svnadmin create stuff
    # # chown -R apache.apache stuff
    # # chcon -R -t http_sys_content_t stuff

    Last line should rather set SELinux context to http_sys_content_rw_t:

    chcon -R -t http_sys_content_rw_t stuff

    But anyway let's set our httpd configuration to be:

    • path for all repositories /var/www/svn

    • for apache authentication I choose the simplest one Basic with password file stored in /var/svn/passwd

    • finally I choose to store global svn authorization file in /var/svn/svnauth


    so using above presettings our subversion.conf would look like:

    LoadModule dav_svn_module modules/mod_dav_svn.so
    LoadModule authz_svn_module modules/mod_authz_svn.so
    <Location /repos>
    DAV svn
    SVNParentPath /var/www/svn
    AuthzSVNAccessFile /var/svn/svnauth
    SSLRequireSSL
    Order deny,allow
    AuthType Basic
    AuthName "Subversion repository"
    AuthUserFile /var/svn/passwd
    Require valid-user
    </Location>
    ## logger for svn: /var/log/httpd/svn_log
    CustomLog logs/svn_log "%t %u %{SVN-ACTION}e" env=SVN-ACTION



  3. apache authentication file is of course produced using htpasswd command:

    # for 1st user we're creating a file (-c) and choose MD5 encryption (-m)
    sudo htpasswd -cm /var/svn/passwd cibak
    New password: xxxxx
    Retype new password: xxxxx
    Adding password for user cibak
    # all the others users only added with the same encryption
    sudo htpasswd -m /var/svn/passwd jack
    ...
    ...

    Once this file is ready we have to set correct SELinux policy:

    sudo chcon -t httpd_sys_content_t /var/svn/passwd


  4. creation of a new repository:

    sudo svnadmin create /var/www/svn/myrepo
    sudo chown -R apache:apache /var/www/svn/myrepo
    sudo chcon -R -t http_sys_content_rw_t /var/www/svn/myrepo



  5. creation of subversion authorization file /var/svn/svnauth in what ever you choose editor (I prefer emacs):

    # cibak has read-write rights to the whole repository, jack could only read
    [myrepo:/]
    cibak = rw
    jack = r
    # but jack is able to write in his own directory
    [myrepo:/jack]
    jack = rw

    Syntax of this file is better described in the Subversion bible.

    Of course ones again we should remember to set correct SELinux policy context:

    sudo chcon -t httpd_sys_content_t /var/svn/svnauth



  6. restarting of httpd deamon is a last step to switch our repo on:

    sudo /etc/init.d/./httpd restart


  7. et voila, our repository is accessible under https://localhost/repos/myrepo URL.

wtorek, 1 czerwca 2010

Fedora 13 on MBP

Fedora 13. Download iso, put into CD drive, choose upgrade, wait for about 40 minutes, reboot, update GPT/MBRin reFit, boot. Whole procedure took me about one hour. Niiice!

But of course it could be harder if you want to install it from scratch. I had Fedora 12 installed already, so it was quick and easy task.

czwartek, 22 kwietnia 2010

Welcome to Fedora 12!

I had tried to install Fedora 6 on my MacBook Pro 2-3 years ago but those days you have to hack to many things inside the kernel and config, that I gave up after one week of struggling. So, I have to use this %$#$%@ MacOSX, which looks like OS for emo teenager. I mean this young true emo kid, who is cutting himself with a pink razor to see his own artificial blood made of ketchup shouting and crying about Miss Spears private and public life. All this fancy jumping and zooming in and out icons and all this very deeply hidden options/configuration files...

Hopefully in the meantime good guys from Fedora were working on a new releases. And once Fedora 12 was released (well, I should say once I have realised it) I had gazed my eyes onto it eager to try it out once again. So I've downloaded x86_64 DVD ISO image, burned it and kept it at my fingertips, but the next step was o download and install in MacOSX small application called rEFIt. Afterwards I've downloaded and burned ISO image of gparted, moved, just for curiosity all my files to external USB hard drive and the repartitioning started straight away... I have shrunk main HD hpfs+ partition to 20 GB and created small /boot (300 MB), bigger /root (30GB) and rather huge /home (the rest) partitions, all using ext4 format. I've also created really small swap (2GB) at the very end of disk.

Then I put Fedora 12 installation DVD into slot and installation started. When it completed new lovely Tux has shown up under rEFIt menu just after resyc'ing HD master boot record with GUID. Then I logged to MacOSX once again only for changing boot options and choosing Fedora by default. I can't remember what I exactly did, but all details are in rEFIt documentation.

And then I've rebooted my laptop once again waiting with my fingers crossed to boot into Fedora 12 for the very first time.

What works out of the box? Built-in camera, track pad, ethernet and wireless cards, graphic adapter, special keys for LCD brightness, sound volume and keyboard backlight.

For the sound I have to tweak it a little putting into
/etc/modprobe.d/sound file this config:

alias snd-card-0 snd-hda-intel
alias sound-slot-0 snd-hda-intel
options snd-hda-intel model=mbp3

and reboot it once.

Then I set up Polish keyboard mapping (System → Preferences → Keyboard → Layout → Add and look for Polish, then modify Options for input of Polish characters, AFAIR I've ticked "Any ALT key" in "Key to choose 3rd level" option). Et voila: ąśźżłóę. :)

Then I went to standard procedure of installation of VERY IMPORTANT APPS (krb5, openafs, LDAP for Thunderbird etc.) googling for their configuration in cern.ch domain.

And that's it! Since one month I'm working on Fedora 12. And I'm very happy!

Post-installation took me some more time, especially that Fedora guys are really good guys and they're not distributing any piece of code which has got some problem with the Law (so i.e. there are no mp3 codecs in official Fedora yum repos etc.). Fortunately all is quite good documented on Internet by not-so-good guys and easy to find thanks to Google, Inc.

Farewell MacOSX! Never see you again! Never ever come back!!!

wtorek, 16 marca 2010

CherryPy behind blue eyes of apache

Well, a big change is coming. We are going to switch our web app from PHP+jQuery to CherryPy+jQuery+some kind of ORM+some db back-end (and with a little intrusion of PHP, but #!).

It won't be easy... First of all I've put all parts of a new system together:

  • CherryPy 3.1.2 for "pythonization"
  • SQLObject 0.10.2 as out ORM
  • Cheetah 2.2.2 for templates
  • sqlite or mysql for db back-end (still not decided)


I want to run CherryPy application in a standalone server, probably as a deamon, on our www host and communicate with it using apache + mod_proxy.


  1. deamonization

    Sound easy but it took me two days to understand cherryd usage. After that I've wrote a shell script to run my CherryPy application as a deamon under Fedora (I hope it would work also under RHEL..):


    #!/bin/sh
    #
    # Comments to support chkconfig on RedHat Linux
    # chkconfig: 35 96 96
    # description: Cherrypy start/stop script
    #
    #
    ### BEGIN INIT INFO
    # Provides: cherrypy
    # Required-Start: $local_fs $syslog
    # Required-Stop: $local_fs $syslog
    # Default-Start: 2345
    # Default-Stop: 90
    # Short-Description: run CherryPy web server
    # Description: CherryPy is a python framerwork for CMT
    #
    ### END INIT INFO

    . /etc/init.d/functions

    serverdir=/home/cibak/CheRTTyPy
    prog=get
    server=${serverdir}/get.py
    config=${serverdir}/conf/get.conf
    pidfile=${PIDFILE-/var/run/CheRTTyPy.pid}
    lockfile=${LOCKFILE-/var/lock/subsys/CheRTTy}

    PYTHONPATH=${serverdir}

    RETVAL=0

    start() {
    echo -n "Starting $prog:"
    cherryd -d -p ${pidfile} -c ${config} -i get
    RETVAL=$?
    [ $RETVAL = 0 ] && touch ${lockfile}
    echo_success
    echo
    return $RETVAL
    }

    stop() {
    echo -n "Stopping $prog:"
    if test -f ${pidfile}; then
    killproc -p ${pidfile} -d 10 $prog
    rm -f "$pidfile"
    RETVAL=$?
    else
    RETVAL=1
    fi
    RETVAL=$?
    echo_success
    echo
    [ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile}
    }


    case "$1" in
    start)
    start
    ;;
    stop)
    stop
    ;;
    status)
    status -p ${pidfile} $server
    RETVAL=$?
    ;;
    restart)
    stop
    start
    ;;
    *)
    echo $"Usage: $0 {start|stop|restart|status}"
    exit 1
    esac
    exit $RETVAL


  2. apache and mod_proxy

    I want to run CherryPy app behind apache and use mod_proxy to redirect ajax calls to it. So the vhost configuration:


    <VirtualHost cibak.fedora12.mbp:443>
    ServerAdmin krzysztof.ciba@xxxxxxxxx.xxx
    ServerName cibak.fedora12.mbp
    ProxyPreserveHost On

    DocumentRoot /home/cibak/CheRTTyPy
    ErrorLog logs/cibak.fedora12.mbp_error.log
    TransferLog logs/cibak.fedora12.mbp_access.log

    SSLEngine On
    SSLProtocol all -SSLv2
    SSLProxyEngine On

    SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW

    SSLCertificateFile /etc/pki/tls/certs/localhost.crt
    SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

    <Proxy *>
    Order allow,deny
    Allow from all
    </Proxy>
    ProxyPass /query/ https://127.0.0.1:8083/
    ProxyPassReverse /query/ https://127.0.0.1:8083/
    </VirtualHost>


  3. ORM
    SQLObject, looks pretty nice, especially sqlbuilder classes for constructing SQL query. For prototype I choose simple sqlite3 DB-API, as it is already built in python, but for production version I'll switch to ORACLE or mysql.

    BTW switch of db back-end is very easy with SQLObject ORM — only place for that is changing connection string passed to sqlconnector. :)


After 6-7 weeks of rather extensive work the prototype for Atlas experiment RTT web page is ready and released to the public. Now I'm waiting for the feedback.



!!!WORK IN PROGRESS!!!

niedziela, 21 lutego 2010

Personas for Firefox

Want to be a better Atlas member? Want to be a cool cat in your office? Want to impress your friends around? Use
this! ;)

wtorek, 9 lutego 2010

reallocate your code from one repository to another

I'm a bit too much python fan, so once again it would be a python script. Sorry! ;)

Imagine you want to move your code from one Subversion repository to another with all history preserved, but under different directory. No problem at all.


  1. First you need to dump contents of, let's say, source repository to the dump file, probably filtering out someone else's stuff:


    $> svnadmin dump /path/to/src/repo | \
    svndumpfilter --drop-empty-revs --renumber-revs \
    --skip-missing-merge-sources include SRCDIR > SRCDIR.dump


  2. Now filtering script:


    ##
    # @file realloc.py
    # @author Krzysztof Daniel Ciba (Krzysztof.Ciba@NOSPAMgmail.com)
    #
    import sys, os
    import optparse
    ##
    # @class realloc
    # @author Krzysztof Daniel Ciba (Krzysztof.Ciba@NOSPAMgmail.com)
    # @brief reads stdin dump file, changes all paths from SRC to DEST and prints it out to stdout
    class realloc( object ):

    ## c'tor
    # @param self "Me, myself and Irene"
    # @param src source path in dump file
    # @param dest destination path on dump file
    def __init__( self, src, dest ):
    self.src = src
    self.dest = dest

    def run( self ):
    for line in sys.stdin:
    line = line.strip("\n")
    if ( "Node-path:" in line or
    "Node-copyfrom-path:" in line ):
    if self.src in line:
    line = line.replace( self.src, self.dest )
    print line

    def check( option, opt_str, value, parser ):
    if ( str(value).startswith("/") ):
    raise optparse.OptionValueError( "value of " + opt_str + " should be a relative path!" )
    setattr( parser.values, option.dest, value )

    ## trigger processing
    if __name__ == "__main__":
    version = "%prog $Revision: 8715 $ by Krzysztof Daniel Ciba (Krzysztof.Ciba@NOSPAMgmail.com)"
    usage = "%prog [opts] [< indumpfile > outdumpfile]"
    parser = optparse.OptionParser( usage=usage, version=version )
    parser.add_option("--src", type="string", action="callback", callback=check, dest="src", help="source directory" )
    parser.add_option("--dest", type="string", action="callback", callback=check, dest="dest", help="destination directory" )
    opts, args = parser.parse_args(sys.argv[1:])
    if ( not opts.src and opts.dest ):
    parser.error("option --dest present but --src is missing")
    elif ( opts.src and not opts.dest ):
    parser.error("option --src present but --dest is missing")
    elif ( not opts.dest and not opts.dest ):
    parser.error( "options --src and --dest are required" )
    else:
    theApp = realloc( opts.src, opts.dest )
    sys.exit( theApp.run() )


  3. and it's action


    $> realloc.py --src SRCPATH --dest DESTPATH < SRCPATH.dump > DESTPATH.dump


  4. and standard loading to new repository:


    $> svnadmin load /path/to/new/repo < DESTPATH.dump




Not so much work, especially it could be used in pipes, i.e. dumping, filtering, reallocating and loading in the same time:


$> svnadmin dump /path/to/src/repo | svndumpfilter \
--drop-empty-revs --renumber-revs --skip-missing-merge-sources \
include SRCDIR | realloc.py --src SRCDIR --dest DESTDIR | \
svnadmin load /path/to/dest/repo


Hmmm... Of course it could be made better in some simple way using sed. Yes, I'm using sed from time to time, but NOT this time. ;)

czwartek, 28 stycznia 2010

hacking cvs2svn ;)

Several months ago my BOSS asked me to transfer our ancient CVS repository to the brand new Subversion system. Not a big deal -I thought - there is lovely tool which does it for me: cvs2svn converter made by Tigris. But just after that I realized it wouldn't be such easy task, as our current repository is rather huge and holds several years of our projects history. Some fraction of packages was obsolete and additionally there was a request to split it to three different SVN repos and keep in a newly Subversion repos only recent year of history.

Not a big deal once again - I said after reading of cvs2svn documentation - we've got a database holding all the cvs tags used to built our software, so I'd make a few queries to know what to keep and that's all, except... OMG! There is no "include" pattern for tags!!!

You could in a easy way exclude symbols from cvs, which shouldn't be converted, but I've got completely opposite situation. Now, smart guys, please try to write me a reqexp which is nagative to a pattern... Oups!

What I did it was a little modification of cvs2svn tool in cvs2svn/cvs2svn_lib/symbol_strategy.py:

Then of course I wrote a script which:

  • asked our tag db for packages and their tags/branches that should be migrated
  • generated cvs2svn option file
  • and run cvs2svn tool


Several testing later a big migration day arrived on agenda. I've started to migrate about 9:00 AM and before 6:00 PM ca. 2k packages were moved generating initially about 190k revisions. And one day later we've opened new repos to the whole collaboration.