Strony

Pokazywanie postów oznaczonych etykietą subversion. Pokaż wszystkie posty
Pokazywanie postów oznaczonych etykietą subversion. Pokaż wszystkie posty

wtorek, 9 lutego 2010

reallocate your code from one repository to another

I'm a bit too much python fan, so once again it would be a python script. Sorry! ;)

Imagine you want to move your code from one Subversion repository to another with all history preserved, but under different directory. No problem at all.


  1. First you need to dump contents of, let's say, source repository to the dump file, probably filtering out someone else's stuff:


    $> svnadmin dump /path/to/src/repo | \
    svndumpfilter --drop-empty-revs --renumber-revs \
    --skip-missing-merge-sources include SRCDIR > SRCDIR.dump


  2. Now filtering script:


    ##
    # @file realloc.py
    # @author Krzysztof Daniel Ciba (Krzysztof.Ciba@NOSPAMgmail.com)
    #
    import sys, os
    import optparse
    ##
    # @class realloc
    # @author Krzysztof Daniel Ciba (Krzysztof.Ciba@NOSPAMgmail.com)
    # @brief reads stdin dump file, changes all paths from SRC to DEST and prints it out to stdout
    class realloc( object ):

    ## c'tor
    # @param self "Me, myself and Irene"
    # @param src source path in dump file
    # @param dest destination path on dump file
    def __init__( self, src, dest ):
    self.src = src
    self.dest = dest

    def run( self ):
    for line in sys.stdin:
    line = line.strip("\n")
    if ( "Node-path:" in line or
    "Node-copyfrom-path:" in line ):
    if self.src in line:
    line = line.replace( self.src, self.dest )
    print line

    def check( option, opt_str, value, parser ):
    if ( str(value).startswith("/") ):
    raise optparse.OptionValueError( "value of " + opt_str + " should be a relative path!" )
    setattr( parser.values, option.dest, value )

    ## trigger processing
    if __name__ == "__main__":
    version = "%prog $Revision: 8715 $ by Krzysztof Daniel Ciba (Krzysztof.Ciba@NOSPAMgmail.com)"
    usage = "%prog [opts] [< indumpfile > outdumpfile]"
    parser = optparse.OptionParser( usage=usage, version=version )
    parser.add_option("--src", type="string", action="callback", callback=check, dest="src", help="source directory" )
    parser.add_option("--dest", type="string", action="callback", callback=check, dest="dest", help="destination directory" )
    opts, args = parser.parse_args(sys.argv[1:])
    if ( not opts.src and opts.dest ):
    parser.error("option --dest present but --src is missing")
    elif ( opts.src and not opts.dest ):
    parser.error("option --src present but --dest is missing")
    elif ( not opts.dest and not opts.dest ):
    parser.error( "options --src and --dest are required" )
    else:
    theApp = realloc( opts.src, opts.dest )
    sys.exit( theApp.run() )


  3. and it's action


    $> realloc.py --src SRCPATH --dest DESTPATH < SRCPATH.dump > DESTPATH.dump


  4. and standard loading to new repository:


    $> svnadmin load /path/to/new/repo < DESTPATH.dump




Not so much work, especially it could be used in pipes, i.e. dumping, filtering, reallocating and loading in the same time:


$> svnadmin dump /path/to/src/repo | svndumpfilter \
--drop-empty-revs --renumber-revs --skip-missing-merge-sources \
include SRCDIR | realloc.py --src SRCDIR --dest DESTDIR | \
svnadmin load /path/to/dest/repo


Hmmm... Of course it could be made better in some simple way using sed. Yes, I'm using sed from time to time, but NOT this time. ;)

czwartek, 28 stycznia 2010

hacking cvs2svn ;)

Several months ago my BOSS asked me to transfer our ancient CVS repository to the brand new Subversion system. Not a big deal -I thought - there is lovely tool which does it for me: cvs2svn converter made by Tigris. But just after that I realized it wouldn't be such easy task, as our current repository is rather huge and holds several years of our projects history. Some fraction of packages was obsolete and additionally there was a request to split it to three different SVN repos and keep in a newly Subversion repos only recent year of history.

Not a big deal once again - I said after reading of cvs2svn documentation - we've got a database holding all the cvs tags used to built our software, so I'd make a few queries to know what to keep and that's all, except... OMG! There is no "include" pattern for tags!!!

You could in a easy way exclude symbols from cvs, which shouldn't be converted, but I've got completely opposite situation. Now, smart guys, please try to write me a reqexp which is nagative to a pattern... Oups!

What I did it was a little modification of cvs2svn tool in cvs2svn/cvs2svn_lib/symbol_strategy.py:

Then of course I wrote a script which:

  • asked our tag db for packages and their tags/branches that should be migrated
  • generated cvs2svn option file
  • and run cvs2svn tool


Several testing later a big migration day arrived on agenda. I've started to migrate about 9:00 AM and before 6:00 PM ca. 2k packages were moved generating initially about 190k revisions. And one day later we've opened new repos to the whole collaboration.

poniedziałek, 25 stycznia 2010

SVN hooks once again, this time tags, branches, log messages etc.

This one hook I wrote quite long ago is forcing users of Subversion repository to:

  • provide log message on every commit (class logChecker)
  • block removing of "/tags", "/branches" and "/trunk" directories from repo (class pathChecker)
  • forcing branch and tag naming convention
  • making subdirectories under /tags read-only


My company is running Subversion for storing all software packages and our repo hasn't got the default structure (/tags, /trunk and /branches as a top directories) — instead of that every package has got it's own /trunk, /tags and /branches subdirectories. So the repository structure is something like this:

/Path/
PackageA/
trunk/
src/
doc/
tags/
PackageA-00-00-00/
src/
doc/
...
branches/
PackageA-00-00-00-branch/
src/
doc/

PackageB/
...
/OtherPath/
PackageC/
...


Looks rather complicated, but it isn't. ;)

Also we've got tags and branches naming convention. Every tag should be made of package name, then a three (or four for tags made over a branch) groups of digits, e.g.: PackageName-ii-jj-kk or PackageName-ii-jj-kk-ll.
Every branch name is very similar to tag name, except there is additional (-branch) string at the end.

So here is the hook itself:


If you want to use this one, you have to modify it a little:

  • replace 'librarian' or/and 'root' account name with your own repository admin
  • put repository admin mail to block messages - replace this fake 'librarian@mail' address
  • change regexp ffor tags and branches to match your criteria
  • put svn-policy.py script into your repository hook directory ($REPO/hooks)
  • modify "$REPO/hooks/pre-commit" script to switch this one on

    #!/bin/sh
    REPOS="$1"
    TXN="$2"
    SVNPOLICY=/path/to/repo/hooks/svn-policy.py # modify this line
    #SVN policy
    $SVNPOLICY -t "$TXN" "$REPOS" || exit 1
    # All checks passed, so allow the commit.
    exit 0



Maybe the code needs some cleaning and better structure but anyway happy using, comments and questions are welcome!

Cheers,
Krzysztof

czwartek, 26 listopada 2009

eol once again

Here is another pre-commit hook I wrote for Subversion repositories. Let's assume many people (not always smart enough) accessing and committing to your Subversion repository. They don't know how to configure their svn client, but you want to be sure, that all text files have the same line endings (e.g. Unix = LF). So, to be sure that all line endings are consistent, you could run this hook, which blocks commit operation if svn:eol-style is not set to required one.

As always you should modify also your pre-commit file, usually stored in repo/hooks directory, e.g.:

And that's it! ;)

poniedziałek, 23 listopada 2009

subversion hook for "syncing" inside one repository

The old school CVS has a very nice ability to make an alias in one module, which points to another one. In Subversion the situation is different, as svn is using it's own "file-system". Of course you can use svn:externals property to pull to your working copy some external source tree required during compilation. But what if you just want for some reason store exactly the same file tree in two different locations in your repo?

For that one you can use post-commit hook, which checks the change list of transaction and if required merge the changes made in one location to another:


#!/usr/bin/env python
##
# @file sync.py
# @author Krzysztof Daniel Ciba (Krzysztof.Ciba@NOSPAMgmail.com)
# @date 06/05/2009
# @brief post-commit hook for syncing two locations in repository
import os, sys
import datetime
import getopt
try:
my_getopt = getopt.gnu_getopt
except AttributeError:
my_getopt = getopt.getopt
import StringIO
from subprocess import Popen
from subprocess import PIPE

twinPaths = { "/destpath/dest1" : "/some/path/src1" }

svnhome = "/PATH/TO/YOUR/REPO/"
svnroot = "file:///PATH/TO/YOUR/REPO"
svnCmd = "/usr/bin/svn"
svnlookCmd = "/usr/bin/svnlook"

## make sync between twin packages
# @param rev revision number
def sync(rev):

svnlook = svnlookCmd + " changed " + svnhome + " -r " + rev
changed = Popen( svnlook.split(), stdout=PIPE).communicate()[0]
changed = StringIO.StringIO( changed )
changed = changed.read()

changed = changed.split( "\n" )

twins = []
for line in changed:
if ( line != "" ):
change, where = line.split()
where = where.strip()
sys.stdout.write( change + " in " + where + "\n" )
for srcPath, dstPath in twinPaths.iteritems():
print where + " " + srcPath

if ( where.startswith( srcPath ) ):
twins.append( ( srcPath, dstPath ) )

noDupes = []
[ noDupes.append(i) for i in twins if not noDupes.count(i) ]
twins = noDupes
if ( len( twins ) ):

for ( srcPath, dstPath ) in twins:
srcURI = svnroot + "/" + srcPath + "@" + rev
dstURI = svnroot + "/" + dstPath

cmd = "#!/bin/sh\n"
cmd += svnCmd + " co " + dstURI + "\n"
cmd += svnCmd + " merge " + dstURI + " " + srcURI + " " + dstPath +"\n"
cmd += svnCmd + " ci " + dstPath + " -m 'syncing " + dstPath + " with " + srcPath + "'\n"
cmd += "rm -rf "+ dstPath +"\n"

sys.stdout.write( cmd )
stdin, stdout, stderr = os.popen3( cmd )
err = stderr.read()
stderr.close()
out = stdout.read()
if ( err != "" ): return False
return True

## write usage and exit
def usage_and_exit(error_msg=None):
import os.path
stream = error_msg and sys.stderr or sys.stdout
if error_msg:
stream.write("ERROR: %s\n\n" % error_msg)
stream.write("USAGE: %s -r REV REPOS\n"
% (os.path.basename(sys.argv[0])))
sys.exit(error_msg and 1 or 0)

## main processing
def main( argv ):
repos_path = None
rev = None
try:
opts, args = my_getopt(argv[1:], 'r:h?', ["help"])
except:
usage_and_exit("problem processing arguments/options")
for opt, value in opts:
if opt == '--help' or opt == '-h' or opt == '-?':
usage_and_exit()
elif opt == '-r':
rev = value
else:
usage_and_exit("unknown option '%s'" % opt)

if rev is None:
usage_and_exit("must provide -r argument")
if len(args) != 1:
usage_and_exit("only one argument allowed (the repository).")

if ( sync(rev) ): return 0
return 1

## start processing
if __name__ == '__main__':
sys.exit( main( sys.argv ) )


Then of course you should modify your post-commit file, in my case it is very simple:


#!/bin/sh
REPOS="$1"
REV="$2"
# synchronisation between packages
SYNC=/PATH/TO/REPO/hooks/sync.py
$SYNC -r "$REV" "$REPOS"


And that's it! Every change (commit) made in /some/path/src1 would be merged and committed to /destpath/dest1.