RIP Spunky

I have known Spunky the dog since my family received him as a puppy 15 years ago. He was always happy and full of energy. I would often catch him smiling :)

You can see him chasing things in an earlier post.

He passed away today :(

Opening a TAR file in Python containing millions of files

I was recently opening at tarball containing millions of XML files. I got about half way through parsing the files and my VM ran out of memory and came to a grinding halt. Something was causing high memory usage. Knowing that Python has automatic memory management I considered the suspects; the tarfile and lxml modules I was using to process the data.

I initially thought this may be because I was not closing the buffer created by the TarFile.extractfile() function. This was not the case.

It turns out that the TarFile class keeps cached copies of information in a variable called members. This is an undocumented workaround that has been documented by Alexander Dutton. The workaround is to set the members variable to an empty list after processing each file.

import tarfile


tar = tarfile.open('large.tar.gz', 'r:gz')
for tarinfo in tar:
  # open the file from the archive as an in memory buffer
  buf = tar.extractfile(tarinfo)
  
  for line in buf:
    # do something with the line
    process(line)
  
  # free the cached data structures held by the TarFile object
  tar.members = []

Alt key not working in Mac OS X terminal

I have had trouble switching windows in irssi (alt + 1, alt +2, ... , alt + n) since I have been using my macbook more often. It turns out that the OS X terminal application rebinds the alt key for shortcuts.

It is possible to rebind the alt key so it will work as expected.

To do this go to,
Terminal > Preferences > Settings > Keyboard

and select 'Use option as meta key' and then the alt key will be passed through to the terminal.

Barebones .vimrc

My barebones .vimrc. Nothing fancy. Just syntax highlighting, auto indenting and tabbing, mouse and an 80 column marker.


syn on
set softtabstop=4
set shiftwidth=4
set tabstop=4
set expandtab
set smarttab
set autoindent
set mouse=a
set textwidth=80
set colorcolumn=+1
hi ColorColumn ctermbg=darkgrey guibg=darkgrey

Spunky Chasing Things

Spunky the dog chasing things.

Bridge to Bridge

I rode along the Brisbane river from the bridge at Indooroopilly to the Gateway bridge and back.

Date:26/06/2010 7:22 am
Distance:65.0 kilometers
Elapsed Time:3:20:22
Avg. Speed:19.5 km/h
Max. Speed:56.3 km/h




View Larger Map

Hyperref and Cleveref LaTeX Conflict

I ran into a conflict between two LaTeX packages when writing my thesis. Both the hyperref and cleveref are useful packages for automatically dealing with references in a document.

Hyperref is a TeX package for making documents with live links in PDF and HTML output formats. This places automatic links for \ref{} and \cite{} commands into the final PDF document. This allows readers to simply click on a reference to a Table, Figure, Equation or Citation and be taken to its location in the document.

The cleveref package enhances LaTeX's cross-referencing features, allowing the format of references to be determined automatically according to the type of reference. This is almost like type inference found in modern programming languages. When using the \ref{} command in LaTeX, one typically mentions the type of reference in the text and then \ref{} provides the number for the reference. For example, Table \ref{table:x2} shows the values of x^2 for integers 1 through 10. Using cleveref one can omit the leading Table and the package automatically infers it from the reference. For example, \Cref{table:x2} shows the values of x^2 for integers 1 through 10.

I used cleveref throughout my entire thesis and I included the hyperref package at the end to add PDF links. Unfortunately this somehow manages to conflict with cleveref and it no longer works as expected. The following demonstrates what happens. The caption of the Figure is inserted into the reference.



I ended up not using the cleveref package for this reason. I am not a LaTeX expert when it comes writing packages, so until I can dedicate some time to learn what is going on I have no working solution.

I used the TeXLive packages from the Ubuntu repository which contains the hyperref package. I downloaded cleveref from the CTAN repository.