Workflow with Subversion and LaTeX

In the previous post I showed how easy it is to install your own Subversion nowadays. But that brings you only that far because you need to integrate SVN into your daily working routine. Here I show how I do that with my LaTeX projects.

The basic workflow with Subversion is as follows:

  • Update the working copy (right-click and ‘SVN Update’). This updates the locally stored file with the one from the repository. You need to do this when working on the project with others.
  • Make some changes to the document.
  • Commit the changes to the repository (right-click and ‘SVN Commit’). Make sure to add some notes what you changed to make version tracking easier.

When working on large documents it is really important for me to visualise changes. MikTeX and TeX Live come with the great tool latexdiff that highlights differences between two documents. The syntax is simply latexdiff old.tex new.tex > diff.tex from the command-line. Using SVN appears perfect for the job as we can access old revisions easily. The syntax is then, for example, latexdiff-vc --svn -r HEAD file.tex, which compares the file.tex with the HEAD revision (i.e. the latest revision in the repository).

I generally work with multi-file LaTeX projects, i.e. the main file is called master.tex where all sections of the project are included via \include or \input (introduction, chapter 1, chapter 2 etc.). For this, latexdiff offers the -flatten option that replaces the \include and \input commands with the actual content. Unfortunately, latexdiff currently does not support -flatten and -vc simultaneously, so we have to do that manually.

The idea is as follows:

  • Flatten the document, i.e. merge the contents of all .tex files into a single document. This includes the bibliography.
  • Compare the flattened document with a previous version with latexdiff and generate a PDF automatically.
  • Commit the flattened document and the PDF.

I also want to create two more versions of the flattened document:

  • One were each sentence starts in a separate line. This is useful if you check differences on the source rather than the compiled document latexdiff changes the source by including \DIFdelbegin and \DIFdel tags to highlight changes, so if you want to copy & paste some of the text later this is more convinient.
  • One were all comments and to-dos (using the todonotes package \todo{}) are removed. This file could be sent to publishers (or just people who want to have a look), but are not supposed to read your comments.

That sounds like a handful, but there are fortunately scripts around for every task I mentioned above. As my programming skills are just at cavemen level I am only able to combine these scripts into a single batch file that does all these steps in one go.

What you need:

  • flatex flattens the document. It is quite old but works perfectly. It also inputs the bibliography so that you truly need only one .tex file for the whole project. Get the single flatten.c source here and compile or download the executable  flatex.exe (1143 downloads) .
  • ltxclean.pl by . A perl script that removes comments and to-dos (see post). In principle it flattens the document as well, but for some reason it messes up the bibliography and tables in my files. So we don’t use that feature and use flatex for flattening.
  • fmtlatex.pl by Andrew Stacey. Another perl script that moves each sentence to a new line.

For the last two scripts to work you need to install Perl.

Now it’s time for my batch  cleandiff.bat (1205 downloads)  that combines all scripts. To run it you need to copy it were your master/main tex-file is (in my script it is called ‘master.tex’). You also need to specify were you save the three scripts above (I just copied them to C:).

Here’s what it does:

Create new working folder. Cleandiff contains all new files, temp will be deleted

mkdir cleandiff
mkdir cleandiff\temp

Call the ancient flatex. It does its job perfectly, but for some reason it does not create a new file as it is supposed to do, but instead the new file is called.flt. So we just rename it to clean-comments.tex and move it to a new folder.

C:\flatex.exe master.tex
rename master.flt clean-comments.tex
move clean-comments.tex cleandiff\clean-comments.tex

cd cleandiff

We add and commit this file which is then the HEAD Revision.

svn add clean-comments.tex
svn commit -m "" clean-comments.tex

Remove comments and \todo{} remarks. On its own the script does the same as flatex, but it does not work well with my documents (tables and the bibliography get messed up). So we just call it, if necessary, after flatex.

perl C:\ltxclean.pl clean-comments.tex > clean-nocomments.tex

Put each sentence into a new line. Useful if using svn diff as it makes it easier to track changes. Should be called upon clean-comments.tex as it is useful to keep tack on your comments and todos as well.

perl C:\fmtlatex.pl clean-comments.tex > clean-linebreak.tex

Compare HEAD revision with the PREVIOUS revision. First checkout, the run latexdiff. You can specify the revisions you want to compare, e.g. -r 28 uses the 28th revision.

svn cat -r PREV clean-comments.tex > temp\PREV.tex
svn cat -r HEAD clean-comments.tex > temp\HEAD.tex

latexdiff -t UNDERLINE temp\PREV.tex temp\HEAD.tex > rev_head_prev.tex

Compile the document. It is necessary to run pdflatex 3 times in that order to get the bibliography and cross references right. nonstopmode is used to ignore errors.

pdflatex rev_head_prev.tex -interaction=nonstopmode
bibtex rev_head_prev.tex
pdflatex rev_head_prev.tex -interaction=nonstopmode
pdflatex rev_head_prev.tex -interaction=nonstopmode

We delete some files that are not required to track changes. The last command deletes empty 0 byte files that may be generated.

del *.lof
del *.lot
del *.toc
del *.log
del *.dvi
del *.aux
del *.bbl
del *.blg
del *.brf
del *.out
rmdir /Q /S temp
for /r %%F in (*) do if %%~zF==0 del "%%F"

Finally we add and commit the remaining files

svn add *
svn commit -m ""
cd..
svn commit cleandiff

If all goes well you should have five committed files in the cleandiff folder:

  • clean-comments.tex: the flattened document that is used for latexdiff
  • clean-linebreak.tex: flattened document including comments where all sentences start on a new line.
  • clean-nocomments.tex: flattened document where comments and to-dos are removed
  • rev_head_prev.pdf and rev_head_prev.tex: PDF and .tex highlighting the changes to the previous version.

After a long working day where I made lot’s of improvements on my papers I run the script so that I can always keep track of my changes.

Downloads

Version Control with Subversion for LaTex & Stata (and more…)

Before starting a large project (such as a PhD) it is crucial to have some form version control system for the text and files you are going to produce. I start a small series of blog posts about how I (hope to) achieve this for LaTeX and Stata .do files using the free version control system Subversion (SVN).

One of the main advantages of Microsoft Word is its brilliant system to track changes and make comments, a thing that LaTeX lacks, so it is always a bit difficult to collaborate and be aware of changes. I have only recently switched to LaTeX because of its ability to integrate result tables from Stata automatically (in my opinion the only big advantage of LaTeX. You can do amazing things in Word as well, but more about that in some future posts), so I started looking into version controlling for LaTeX projects. I was always hesitant to try out Subversion because it requires setting up an Apache server, something I had horrible experiences with in the past under Windows, but I have discovered uberSVN by WANdisco, a free and preconfigured Subversion server system – all you have to do is install it like a normal Windows programme. The whole process of setting up an SVN should now take less than 20 minutes (if you are slow).

If you are unsure what Subversion actually is and does, I recommend a look into the LaTeX Wiki. In a nutshell, it allows you to keep track, compare, revise and merge different versions of the same file. It is absolutely crucial if you work on the same project with several authors, but even when you are working alone it is extremely useful. Imagine your supervisor tells you to delete one paragraph. You promptly do so, but the next day (s)he wants it back. You have probably saved the updated document under a new filename, or haven’t, but this system of version control get’s cumbersome after a while, even for small projects. This is where Subversion jumps in. Every time you do a (major) change, you ‘check-in’ in the revised version and SVN stores it under a new revision. If you want to go back, you simply ‘check-out’ the older revision.

I will post more about the workflow in one of the next post, but now let’s have a little ‘Installing Subversion for Dummies’ tutorial. One last piece of advice: Subversion is not a backup tool. If you lose the repository (the database where your files are stored), you lose everything except your local files. For Backup purposes you could use something like Crashplan or Dropbox (that’s what I do).

Installation

  • Download uberSVN. That is the Subversion server. While you are at it, make sure to download the Subversion 1.7.x Client because we will be using the 1.7 version of Subversion which cannot be accessed with older clients.
  • Install uberSVN. I am not going to guide you through the installation process because it is so simply. Make a note of the URL (the default should be 127.0.0.1:9890/ubersvn). If you cannot access the server make sure to restart the browser or your computer. After that install the Subversion 1.7.x Client
  • Now open the control panel in your browser, set up a password etc. Once you are asked where to store the SVN files, choose something like C:\SVN. This folder contains the repository, so it should be nowhere near your locally stored project files.
  • Switch to Subversion 1.7 under Administration > SVN Switch.
  • Now install TortoiseSVN (the link is provided on the dashboard, or download it here). That’s the interface that allows you to manage your files, rather then doing it with a command-line tool. A restart might be required to see the overlays that indicate the file status in Windows Explorer.

Setting up your Data

Now we are going to set up the repository, i.e. the database to store your data. If you have existing data that you want to check-in, i.e. a project in working progress, make sure to follow these steps closely. Most importantly: make a backup of your project files!

  • Set up the repository in uberSVN > Repositories > Add.
  • Give your repository a snappy name, i.e. ‘research’. The URL of the repository will then be, for instance, http://127.0.0.1:9800/research
  • Don’t change anything on the import or permission screens, just click save. The server will then restart and the repository is set up.
  • Switch to Windows Explorer and go to the folder where your original data is stored (i.e. your research/project folder). Right-click on it and select TortoiseSVN > Import. This copies the whole folder into the repository. Enter the URL of your repository (e.g. http://127.0.0.1:9800/research).
  • If you want to make sure that everything is copied click on ‘include ignored files’, as by default Tortoise ignores certain (often useless) file-types. After importing you can delete the folder (again, make a backup).
  • This whole process is necessary so that SVN knows which folders to watch. Therefore go to the folder where you want your project files now to be stored. Right-click on empty space in the Explorer and select SVN Checkout. This copies the content of the repository, i.e. what you just imported, to the new location on your hard-drive.

That’s it! Now Subversion is set up and your project files are being watched. Note that you always have to manually check-in any changes you made to the files, otherwise they are not updated in the repository. If you have have changed a file or folder, this is indicated by a red exclamation mark overlay in the Explorer. To update, just right-click on the item and select SVN Commit.

A version control system is only as good as it’s weakest link: the user. So I will be posting more about the workflow with LaTeX and Stata soon.

New Paper: ‘Self-Control, Financial Literacy and the Co-Holding Puzzle’

J. Gathergood and I have released a new paper titled ‘Self-Control, Financial Literacy and the Co-Holding Puzzle’. It is concerned with the empirical phenomenon that a substantial proportion of the population hold liquid savings and unsecured credit simultaneously – and not small sums, but in the region of £6,000 or more. It is available on SSRN and the abstract tells you more:

We use UK household survey data incorporating measures of financial literacy and behavioural characteristics to analyse the puzzling co-existence of high cost revolving consumer credit alongside low yield liquid savings in household balance sheets, which we term the ‘co-holding puzzle’. Approximately 20% of households in our sample co-hold, on average, £6,500 of revolving consumer credit alongside £8,000 of liquid savings. Co-holders are typically more financially literate, with above average income and education. However, we show co-holding is also associated with impulsive spending behaviour on the part of the household. Our results lend empirical support to theoretical models in which sophisticated households co-hold as a means of managing a self-control problem.

So why do these households not simply payoff their credit and save a substantial amount of money? The intuitive answer is that households want to keep cash as a measure of last resort, for example for (unexpected) expenditure that cannot be paid off using credit. This paper by Telyukova mentions medical and housing expenses that cannot be paid off using credit card. But is it reasonable to assume that households expect to pay off these large sums in the future? Also, Telyokova’s argument focuses on credit card co-holding, but in the UK consumers hold complex credit portfolios, including payday loans, car loans, overdrafts etc. These forms of credit can be used to pay off almost all expenditure, and are readily available.

We focus on another explanation: we find that households that are more likely to co-hold are typically more sophisticated, i.e. have larger incomes and higher financial literacy, but score high on a self-reported measure on impulsiveness. The intuition is that they anticipate impulsive spending, but – because they are sophisticated – deliberately hold outstanding consumer credit balances so as to limit the spending possibilities.

Welcome!

Welcome to my new Webpage!

On this website I will blog about my research, topics in economics in general and I will give some tips and tricks related to digital technology and academic research, such as reference management, Stata, LaTeX etc.

You can follow me on Twitter (jwe_cc) and you can find more about me on the About Me page.