git-annex tries to ensure that the configured number of copies of your
data always exist, and leaves it up to you to use commands like git annex
get
and git annex drop
to move the content to the repositories you want
to contain it. But often, it can be good to have more fine-grained
control over which content is wanted by which repositories. Configuring
this allows the git-annex assistant as well as
git annex get --auto
, git annex drop --auto
, git annex sync --content
,
etc to do smarter things.
Preferred content settings can be edited using git
annex vicfg
, or viewed and set at the command line with git annex wanted
.
Each repository can have its own settings, and other repositories will
try to honor those settings when interacting with it.
(So there's no local .git/config
for preferred content settings.)
The idea is that you write an expression that files are matched against. If a file matches, the repository wants to store its content. If it doesn't, the repository wants to drop its content (if there are enough copies elsewhere to allow removing it).
finding preferred content
To check at the command line which files are matched by preferred content settings, you can use the --want-get and --want-drop options.
For example, git annex find --want-get --not --in .
will find all the
files that git annex get --auto
will want to get, and git annex find
--want-drop --in .
will find all the files that git annex drop --auto
will want to drop.
writing expressions
[[!template Error: failed to process template note template note not found ]]
The expressions are very similar to the matching options documented on the git-annex-matching-options man page. At the command line, you can use those options in commands like this:
git annex get --include='*.mp3' --and -'(' --not --largerthan=100mb -')'
The equivalent preferred content expression looks like this:
include=*.mp3 and (not largerthan=100mb)
So, just remove the dashes, basically. But, there are some differences between the command line options and expressions, so see the documentation below to get the full story.
include=glob
andexclude=glob
Match files to include, or exclude.
While --include=glob and --exclude=glob match files relative to the current directory, preferred content expressions always match files relative to the top of the git repository.
For example, suppose you put files into
archive
directories when you're done with them. Then you could configure your laptop to prefer to not retain those files, like this:exclude=*/archive/*
copies=number
Matches only files that git-annex believes to have the specified number of copies, or more. Note that it does not check remotes to verify that the copies still exist.
To decide if content should be dropped, git-annex evaluates the preferred content expression under the assumption that the content has already been dropped. If the content would not be wanted then, the drop can be done. So, for example,
copies=2
in a preferred content expression lets content be dropped only when there are currently 3 copies of it, including the repo it's being dropped from. This is different than runninggit annex drop --copies=2
, which will drop files that currently have 2 copies.copies=trustlevel:number
Matches only files that git-annex believes have the specified number copies, on remotes with the specified trust level. For example,
copies=trusted:2
To match any trust level at or higher than a given level, use
trustlevel+
. For example,--copies=semitrusted+:2
copies=groupname:number
Matches only files that git-annex believes have the specified number of copies, on remotes in the specified group. For example,
copies=archive:2
Preferred content expressions have no equivalent to the
--in
option, but groups can accomplish similar things. You can add repositories to groups, and match against the groups in a preferred content expression. So rather than--in=usbdrive
, put all the USB drives into a "transfer" group, and usecopies=transfer:1
lackingcopies=number
Matches only files that git-annex believes need the specified number or more additional copies to be made in order to satisfy their numcopies settings.
approxlackingcopies=number
Like lackingcopies, but does not look at .gitattributes annex.numcopies settings. This makes it significantly faster.
inbackend=name
Matches only files whose content is stored using the specified key-value backend.
inallgroup=groupname
Matches only files that git-annex believes are present in all repositories in the specified group.
smallerthan=size
andlargerthan=size
Matches only files whose content is smaller than, or larger than the specified size.
The size can be specified with any commonly used units, for example, "0.5 gb" or "100 KiloBytes"
metadata=field=glob
Matches only files that have a metadata field attached with a value that matches the glob. The values of metadata fields are matched case insensitively.
To match a tag "done", use
metadata=tag=done
To match author metadata, use
metadata=author=* Smith
present
Makes content be wanted if it's present, but not otherwise.
This leaves it up to you to use git-annex manually to move content around. You can use this to avoid preferred content settings from affecting a subdirectory. For example:
auto/* or (include=ad-hoc/* and present)
Note that
not present
is a very bad thing to put in a preferred content expression. It'll make it want to get content that's not present, and drop content that is present! Don't go there..inpreferreddir
Makes content be preferred if it's in a directory (located anywhere in the tree) with a particular name.
The name of the directory can be configured using
git annex enableremote $remote preferreddir=$dirname
(If no directory name is configured, it uses "public" by default.)
standard
git-annex comes with some built-in preferred content expressions, that can be used with repositories that are in some standard groups.
When a repository is in exactly one such group, you can use the "standard" keyword in its preferred content expression, to match whatever content the group's expression matches. (If a repository is put into multiple standard groups, "standard" will match anything.. so don't do that!)
Most often, the whole preferred content expression is simply "standard". But, you can do more complicated things, for example:
standard or include=otherdir/*
groupwanted
The "groupwanted" keyword can be used to refer to a preferred content expression that is associated with a group. This is like the "standard" keyword, but you can configure the preferred content expressions using
git annex groupwanted
.Note that when writing a groupwanted preferred content expression, you can use all of the keywords listed above, including "standard". (But not "groupwanted".)
For example, to make a variant of the standard client preferred content expression that does not want files in the "out" directory, you could run:
git annex groupwanted client "standard and exclude=out/*"
Then repositories that are in the client group and have their preferred content expression set to "groupwanted" will use that, while other client repositories that have their preferred content expression set to "standard" will use the standard expression.
Or, you could make a new group, with your own custom preferred content expression tuned for your needs, and every repository you put in this group and make its preferred content be "groupwanted" will use it.
For example, the archive group only wants to archive 1 copy of each file, spread among every repository in the group. Here's how to configure a group named redundantarchive, that instead wants to contain 3 copies of each file:
git annex groupwanted redundantarchive "not (copies=redundantarchive:3)" for repo in foo bar baz; do git annex group $repo redundantarchive git annex wanted $repo groupwanted done
unused
Matches only keys that
git annex unused
has determined to be unused.This is related the the --unused option. However, putting
unused
in a preferred content expression doesn't make git-annex consider those unused keys. So when git-annex is only checking preferred content expressions against files in the repository (which are obviously used),unused
in a preferred content expression won't match anything.So when is
unused
useful in a preferred content expression?- Using
git annex sync --content --all
will operate on all files, including unused ones, and takeunused
in preferred content expressions into account. - The git-annex assistant periodically scans for unused files, and moves them to some repository whose preferred content expression says it wants them. (Or, if annex.expireunused is set, it may just delete them.)
- Using
anything
Matches any version of any file.
not expression
Inverts what the expression matches. For example,
not include=archive/*
is the same asexclude=archive/*
and
/or
/( expression )
These can be used to build up more complicated expressions.
upgrades
It's important that all clones of a repository can understand one-another's preferred content expressions, especially when using the git-annex assistant. So using newly added keywords can cause a problem if an older version of git-annex is in use elsewhere.
Before git-annex version 5.20140320, when git-annex saw a keyword it did not understand, it defaulted to assuming all files were preferred content. From version 5.20140320, git-annex has a nicer fallback behavior: When it is unable to parse a preferred content expression, it assumes all files that are currently present are preferred content.
Here are recent changes to preferred content expressions, and the version they were added in.
- "anything" 5.20150616
- "standard" 5.20140314
(only when used in a more complicated expression; "standard" by itself has been supported for a long time) - "groupwanted=" 5.20140314
- "metadata=" 5.20140221
- "lackingcopies=", "approxlackingcopies=", "unused=" 5.20140127
- "inpreferreddir=" 4.20130501
Is there a way to drop only the files that are located in an "archive" directory? I want to drop all files when calling
if I move them to the archive. But I want to keep the files that are outside of the archive, even if they are already present in other repos. As far as I have seen and tested, as soon as I have the files in an other repo all files get dropped, also the ones outside the archive directory. Or do I have to increase "numcopies" in order to circumvent the "(not copies=semitrusted+:1)" case?
The expressions used for "standard" are built in.
But, you can use "groupwanted" instead, see documentation above.
Is there a way to change these definitions for a given annex?
ie: in this repo make "client" mean
Built a new copy of git-annex yesterday. I have a "client" on my macbook, and two "backup"s, one on an external HD, one on an ssh git remote.
git annex get --auto works beautifully!
It doesn't seem to work for copying content to a place where it's needed, though.
If I drop a file from my "backup" USB drive, and then go back to my macbook and do a "git annex sync" and "git annex copy --to=usbdrive --auto" it does not send the file out to the USB drive, even though by preferred content settings, the USB drive should "want" the file because it's a backup drive and it wants all content.
Similarly, if I add a new file on my macbook and then do a "git annex copy --to=usbdrive auto" it does not get copied to the USB drive.
Is this missing functionality, or should the preferred content setting for remotes only affect the assistant?
drop --auto
will only drop files that are not preferred content. I'd need to know what preferred content expression you're using to say more.How does the preferred content settings interfere with the numcopies setting?
I could not get behind it. E.g. a case I do not unterstand:
I have a preferred setting evaluating to true and still
does nothing, if the number of copies produced would surpass the numcopies setting.
Thx