Developing TM

From tm5
Revision as of 09:13, 16 January 2015 by Pls (Talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search



Before you start, you should get familiar with the TM5 version control strategy and the repository layout.

If you use (t)csh instead of bash/ksh/sh, then replace "repo=..." with "set repo ..." in the following instructions.

There are several ways of developing TM. You can:

Editing the trunk of a project (or the base) should be limited to small edits if you are not the administrator/owner of that project (see Administrators), and you should agree with the administrator/owner before committing any changes to his/her trunk.

To illustrate the examples below, we assume that you have or want to install TM5 in your $HOME and we use the following tree:

 |-- base
 |    |-- branches
 |    |-- release
 |    `-- trunk
 `-- proj
      `-- MyProject
          |-- branches
          |    `-- MyFancyBranch
          |         |-- py
          |         |-- rc
          |         `-- src
          |-- release
          `-- trunk
               |-- py
               |-- rc
               `-- src

Finally, have a look at the SVN Red Book for more detailed information.

Creating a new project

When you want to significantly modify the code that goes on top of the base, it may be better to start fresh with a new project instead of branching out of an existing one. A typical situation occurs when you want to redefine the tracers for example.

New projects go below the TM5/proj/ dir. Note that the tree structure below the base:

  |-- branches
  |-- release
  `-- trunk

is used as needed below each project; there is no code outside it. You probably only need trunk to start with. Release directories hold frozen copies of the trunk, and are discussed in Making releases. Branches are usually created for major editing of a trunk, and discussed below.

Finally, all files in the trunk must be in one of these sub-directories:

 |-- bin   # scripts
 |-- py    # python scripts
 |-- rc    # configuration file
 `-- src   # code source

You only need the src directory to start. So, a new TM project starts with creating a project/trunk/src tree.

Before creating a new project

You must first know if your ~/TM5/proj (if you have one) is under revision control. Just check if there is a .svn sub-directory:

    cd ~/TM5/proj
    ls -a .svn

A versioned directory will gives something like:

   .  ..  all-wcprops  entries  prop-base  props  text-base  tmp

A not versioned directory will give something like:

   ls: cannot access .svn: No such file or directory

Local creation

If your TM5/proj is under revision control, you can work locally:

cd ~/TM5/proj
svn mkdir --parents MyProject/trunk/src

Then commit your changes to the repository (do not skip that step!):

svn commit -m "[MyProject] created my brand new project"

Now you can add your project to the list of sources in your main rc file (see Building and compiling TM5).

Remote creation

If your TM5/proj is not under revision control (or haven't one), you just create in the repository and then checkout:

  svn -m "created my brand new project" --parents mkdir ${repo}/MyProject/trunk/src
  svn checkout ${repo}/MyProject $HOME/TM5/proj/MyProject

Note: if you try this method in a versioned directory, you should not call svn checkout. Instead execute: svn up.

Creating a new branch

A project (or the base) typically has a trunk and maybe one or more branches in the branches/ sub-directory. Before inclusion in the trunk, major modifications to a project are done in a branch. A branch starts as a copy of the trunk under the branches directory.

It is recommended to work in the repository. So create your branch remotely:

 svn copy  ${repo}/MyProject/trunk  ${repo}/MyProject/branches/MyFancyBranch -m "created MyFancybranch"

Then there are two scenarios to get a working copy, depending on your TM installation:

(1) If you have no local copy of the project, do a checkout:

 svn checkout ${repo}/MyProject  ~/TM5/proj/MyProject

(2) If you already have the project, simply update it:

 cd ~/TM5/proj/MyProject
 svn up

Once you have created your branch, you can add it to the list of TM sources in your main rc file (see Building and compiling TM5), and start modifying its code.


There are two situations, where you want to synchronize your branch and its trunk:

Merging from trunk to branch

To get trunk modifications into your branch follow these steps:

 cd ~/TM5/proj/MyProject/branches/MyFancyBranch
 svn ci -m "latest changes"    # you need a clean branch
 svn merge  ${repo}/MyProject/trunk
 # once you are satisfied with the new code 
 svn ci -m "merged trunk changes into my branch"

Note that this works because the branch started as a copy of the trunk. If you want to repeat the operation, you must update your branch for merge to work (svn up).

Merging from branch to trunk

The operation is almost similar. You essentially switch trunk and branch, but:

 cd ~/TM5/proj/MyProject/trunk
 svn up    # you need an up-to-date trunk
 svn --reintegrate merge  ${repo}/MyProject/branches/MyFancyBranch   # the --reintegrate flag is needed if your svn version is older than 1.8
 # once you are satisfied with the new code 
 svn ci -m "merged my fancy branch into the trunk"
 # clean up (not necessary for svn 1.8+)
 svn delete ${repo}/MyProject/branches/MyFancyBranch -m "remove unusable branch"
 cd ..
 svn up

Note that, if your svn version is not 1.8 or above, you must use the --reintegrate flag, and delete the branch (you can always recreate it later for another round of development). If you have svn 1.8, you can keep the branch and continue developing.

Using subversion with your new branch/project


Help on svn :

svn help

And help on a specific command CMD:

svn help CMD

which is particularly useful to see all available options.

Basic work flow

While developing your own TM project, using the repository offers the advantages of a controlled version system ("information is never lost"). You just need few commands to get going once you have created your own branch or project. The simple update-edit-commit work flow is likely to be enough for most users. To update (up) a directory DIR, and get the changes that others have committed:

cd DIR
svn up

After editing your code, you commit (ci), ie check-in. To include new files, you first need to schedule them for addition (idem for file you want to remove):

svn add FILE-TO-ADD
svn ci -m "my message"

Examining the code

The status, log, and diff commands can provide lots of information about your code. The log gives a commit history. The status tells you if you modified the code, if it was merged, or scheduled for deletion/addition, ...

svn log FILE
svn status
svn -u status
svn -uv st

The diff command is a powerful. To have an overview of your edits, i.e. the differences between your code and the version you checked-out or last updated to:

svn diff [file]      ; show your edits w/r/t to the version you last checkout/updated (known as BASE)

The command let you compare any revisions (the following may require access to the repository):

svn diff -r HEAD [file]   ; show difference between your copy and the most recent (HEAD) in the repos.
svn diff -r PREV [file]   ; show last change as recorded in the repository w/r/t to the BASE.

Undo your edits

Once in a while, you just want to forget all your (uncommitted) edits and revert to a pristine version of the code. The revert command let you do that. To disregard all edits :

svn revert file      ; remove all your changes, revert to BASE (the last revision you checked-out/updated-to)

Temporarily Rollback

You can also return to an older version of the code. This is not a true rollback: the repository does *not* record the action, and remains untouched. But your copy is just going back in time.

To put the file user_output_mmix.F90 back to a state before revision #4551, just update to the revision #4550:

svn update -r 4550 user_output_mmix.F90

then to smoothly revert to the current HEAD (latest revision in the repository), do no delete the file, just do:

svn up

True rollback

A true rollback will undo a previous commit, and record it in the repository. Its application can be restricted to a file instead of the entire current directory. To undo commit #345 for file FILE:

svn merge -c -345 FILE
svn ci -m "undo rev #345"

Making a release

A release is just a copy of the trunk into a branch. It is a convenient frozen copy: no commit should be made to a release branch. Here is a simple script used to make release 4.0:



svn copy  ${repo}/base/trunk  ${repo}/base/release/4.0 -m "release 4.0 base"

svn copy  ${repo}/proj/chem/base/trunk  ${repo}/proj/chem/base/release/4.0 -m "release 4.0 chem/base"

svn copy  ${repo}/proj/user_output/trunk  ${repo}/proj/user_output/release/4.0 -m "release 4.0 user_output"

svn copy  ${repo}/proj/budget/lat10xml/trunk  ${repo}/proj/budget/lat10xml/release/4.0 -m "release 4.0 budget/lat10xml"

svn copy  ${repo}/proj/levels/ml91/trunk  ${repo}/proj/levels/ml91/release/4.0 -m "release 4.0 levels/ml91"
svn copy  ${repo}/proj/levels/ml91/tropo34/trunk  ${repo}/proj/levels/ml91/tropo34/release/4.0 -m "release 4.0 levels/ml91/tropo34"

svn copy  ${repo}/proj/levels/ml62/trunk  ${repo}/proj/levels/ml62/release/4.0 -m "release 4.0 levels/ml62"
svn copy  ${repo}/proj/levels/ml62/tropo31/trunk  ${repo}/proj/levels/ml62/tropo31/release/4.0 -m "release 4.0 levels/ml62/tropo31"

svn copy  ${repo}/proj/levels/ml60/trunk  ${repo}/proj/levels/ml60/release/4.0 -m "release 4.0 levels/ml60"
svn copy  ${repo}/proj/levels/ml60/tropo34/trunk  ${repo}/proj/levels/ml60/tropo34/release/4.0 -m "release 4.0 levels/ml60/tropo34"

SVN Issues

Using SVN on ECMWF computers

On the ECMWF systems, add the following lines to the file "~/.subversion/servers" :

 ### from ECMWF advisory desk:
 http-proxy-host =
 http-proxy-port = 3333

SSL handshake failed

If you get an error like this:

 SSL handshake failed: SSL error: A TLS warning alert has been received. (

It is because of "a problem with the svn.knmi. server's SSL configuration since its upgrade to Red Hat 6 and colocation of on the same machine". The workaround:

here is a workaround to circumvent SSL/TLS entirely by using HTTP instead 
of HTTPS. Be aware that this will transfer your password over the
network in plain text:

Move into the working directory where you have your checked out working copy.

svn info

to find the url of your project. Then do the following:

svn switch --relocate

Now you should be able to commit, update, etc...

If you do not have a working copy, just use 'http' as the prefix instead of https to checkout:

svn checkout

Cannot cleanup

It happens that, when trying to commit changes or update a directory, you find your copy locked. For example:

svn: Working copy '/gpfs/h03/krol/SVN/TM5/base/branches' locked
svn: run 'svn cleanup' to remove locks (type 'svn help cleanup' for details)

Then it is possible that svn cleanup does not work! Solutions have been suggested on this thread. Removing the lock may be the easiest solution:

rm -f .svn/lock
svn up

Svn working copy is too old

If you upgrade your svn client to 1.7, after having checkout the code with an older version, you will have an error message like this:

Please see the 'svn upgrade' command
Working copy "~/path/to/repository" is too old (format 10, created by Subversion 1.6)

The quick fix is to upgrade your copy:

cd ~/TM5
svn upgrade
Personal tools