Recent comments posted to this site:

comment 23 70dcb7e7ffdd14351adaf4c40ee7fdd0
[[!comment Error: unsupported page format hs]]
Tue Jul 19 16:49:40 2016

I want to use a recent git-annex version, and I prefer a Docker solution to a binary. There wasn't much documentation on how to do that, so here are the steps I took, just in case anybody finds this information useful. This is not a complete guide, just more than I was able to find so far.

  • Install Docker so that you can access it (eg, docker info) without sudo. (Beyond the scope in here.)

  • Create a git-annex image as follows:

Dockerfile:

FROM debian:unstable
RUN apt-get update && apt-get install -y ssh git man git-annex=6.20160511-1
# maybe install gpg2, other remotes, etc
#RUN apt-get install -y gnupg2 && ln -s /usr/bin/gpg2 /usr/local/bin/gpg
#RUN apt-get install -y golang-go && GOPATH=/usr/local go get github.com/encryptio/git-annex-remote-b2
VOLUME /data
WORKDIR /data

Build image and run basic test:

docker build -t git-annex-6.20160511-1 .
docker run -it --rm git-annex-6.20160511-1 git-annex version
  • You still need to:

    • On host: Replace git-annex by a script that invokes the Dockerized version.
    • In container:
      • Use non-root uids, otherwise you end up creating files with uid 0.
      • Access ssh and gpg credentials on host.
      • Access ~/.gitconfig on host.
      • Mount git repository root as a volume, not just the current dir.
      • Access AWS/B2 credentials, if applicable.

Here is a sample script that achieves these. Name it git-annex, and place it in your PATH before the host git-annex (or just uninstall the latter).

#!/bin/bash
CONT_NAME=${CONT_NAME:-git-annex-6.20160511-1}
# if in git repo, mount root as /data, and cd into relative subdir
# if not, mount cwd as /data
abs_dir=$(readlink -e .)
root_dir=$(git rev-parse --show-toplevel 2>/dev/null || true)
root_dir=${root_dir:-$abs_dir}
rel_dir=${abs_dir#$root_dir}
# if run by git, assume command is git-annex
# otherwise, don't assume, to allow other uses
cmd=
! [ "$(basename "$(readlink -e /proc/$PPID/exe)")" = "git" ] || cmd=git-annex
exec docker run -it --rm \
    -u $(id -u):$(id -g) \
    -v /etc/passwd:/etc/passwd:ro \
    -v $HOME/.ssh:$HOME/.ssh \
    -v $HOME/.gnupg:$HOME/.gnupg \
    -v $HOME/.gitconfig:$HOME/.gitconfig \
    -v "$root_dir":/data \
    ${AWS_ACCESS_KEY_ID:+-e AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID"} \
    ${AWS_SECRET_ACCESS_KEY:+-e AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY"} \
    ${B2_ACCOUNT_ID:+-e B2_ACCOUNT_ID="$B2_ACCOUNT_ID"} \
    ${B2_APP_KEY:+-e B2_APP_KEY="$B2_APP_KEY"} \
    -w /data"$rel_dir" \
    $CONT_NAME $cmd "$@"
Comment by matei.david Fri Jun 3 18:48:11 2016

Actually you do want to exclude "cover.jpg" from being annexed because it will allow you to view the covers of all your books without getting them. Keeping metadata.opf out of annex is not that important because you cannot safely edit the metadata without getting the books first.

It would be great to teach Calibre to see broken symlinks as normal zero length files and work with them accordingly. And we do need a Calibre plugin with buttons "annex get" and "annex unlock".

Comment by Dilyin Thu Jun 2 18:10:33 2016

Looks like it's not possible to set the annex cost of the "web" and "bittorrent" special remotes annex cost.

This doesn't seem to work:

[remote "web"] annex-cost = 500

Perhaps, because these remotes don't actually exist.

Setting annex cost for a webdav remote works but is incremented by 50 for some reason.

Comment by Dilyin Wed Jun 1 18:44:15 2016

Hmm, guyz? Are you serious with these scripts?

  1. git rm -r --cached large_files

    files are indexed as both removed and untracked and are still in place

  2. commit the changes

    files are seen as untracked

  3. git annex add large_files

    files are replaced with symlinks and are in the index

  4. commit changes again

Make sure that you don't have annex.largefiles settings that would prevent annexing the files.

Comment by Dilyin Wed Jun 1 18:36:40 2016

The hash directories for a key can be looked up by passing the key to git annex examinekey --format '${hashdirlower}\n'

Comment by joey Tue May 31 16:18:45 2016

Format appears to be /BASE_DIR/xxx/yyy/FILEKEY/FILEKEY

How can I get the xxx/yyy or the complete storage path of an annexed file in a fast way?

Background:
I have to "chmod" the xxx/yyy within an '.git/hooks/pre-commit' action (shared used NFS location).
Following way using 'find' needs too much time in case of a big remote:

$ git annex info --fast inputbinaries | grep 'directory:' | awk '{print $2}'
/gitworkspace/inputbinaries/DWDM1830-all_playground

$ git annex info vendor-libs/x86/libsnmpdm.a | grep 'key:' | awk '{print $2}'
SHA256E-s3992448--82604383354b28f8efcdf8d83bb63607a8d5967551ded055e782ac342bb7f8e0.a

$ find /gitworkspace/inputbinaries/DWDM1830-all_playground -type f -name SHA256E-s3992448--82604383354b28f8efcdf8d83bb63607a8d5967551ded055e782ac342bb7f8e0.a
/gitworkspace/inputbinaries/DWDM1830-all_playground/28e/785/SHA256E-s3992448--82604383354b28f8efcdf8d83bb63607a8d5967551ded055e782ac342bb7f8e0.a/SHA256E-s3992448--82604383354b28f8efcdf8d83bb63607a8d5967551ded055e782ac342bb7f8e0.a
Comment by uwe.nagler Tue May 31 11:51:56 2016

Please open todo items for patches, don't send them as comments here.

(I suspect that the patch as provided might break compilation with old versions of ghc, or old versions of yesod..)

Comment by joey Fri May 27 15:45:49 2016

I have a patch:

https://github.com/ggreif/git-annex/tree/patch-1

It heals e.g.

Assistant/WebApp/Form.hs:52:1: warning: [-Wredundant-constraints]
    ? Redundant constraint: Monad m
    ? In the type signature for:
           withNote :: (Monad m, ToWidget (HandlerSite m) a) =>
                       Field m v -> a -> Field m v
Comment by ggreif Fri May 27 14:21:11 2016

You can find out the size of a key by using git-annex examinekey $key --format='${bytesize}\n' (There's a --batch option to avoid needing to spin up repeated such processes.)

I suppose I could add a PROGRESSPRECENT, but any version of git-annex that didn't support it would fail with a protocol error if a special remote tried to use that. So, the special remote would need to check git-annex version to use it.

So, maybe better to get the size of the key yourself, and convert the percentage to bytes for PROGRESS.

Comment by joey Mon May 23 18:54:35 2016