---
title: "Announcing fc"
description: "NixOS manages dependencies in a very strict way — sometimes too strict? Here at Flying Circus, many users prefer to compile custom applications in home directories."
language: en
date: 2024-04-02
canonical: "https://flyingcircus.io/en/about-us/blog-news/details-view/announcing-fc-userscan"
format: markdown
generator: "TYPO3 LLMs.txt Extension"
---

# Announcing fc

> NixOS manages dependencies in a very strict way — sometimes too strict? Here at Flying Circus, many users prefer to compile custom applications in home directories.

# Announcing fc-userscan

  Created by Christian Kauhaus |     10/04/2017  |  Blog

 NixOS manages dependencies in a very strict way — sometimes too strict? Here at Flying Circus, many users prefer to compile custom applications in home directories.

 They link them against libraries they have installed before by *nix-env*. This works well... until something is updated! On the next change *anywhere* down the dependency chain, libraries get new hashes in the Nix store, the garbage collector removes old versions, and user applications break until recompiled. In this blog post, I would like to introduce [*fc-userscan*](https://github.com/flyingcircusio/userscan). This little tool scans (home) directories recursively for Nix store references and registers them as *per-user* roots with the garbage collector. This way, dependencies will be protected even if they cease to be referenced from "official" Nix roots like the *current-system* profile or a user's local Nix profile. After registering formerly unmanaged references with *fc-userscan*, one can fearlessly run updates and garbage collection. This problem would not be there if everyone would just write Nix expressions for everything. This is, for various reasons, not realistic. Some users find it hard to accommodate with Nix concepts ("I just want to compile my software and don't care about your arcane Linux distro!"). Others use deploy tools like [zc.buildout](http://docs.buildout.org/) which are explicitly designed for in-place updates and don't really distinguish compile time from run time. With *fc-userscan*, you can get both: Nix expressions for superb reproducibility, and an acceptable degree of reliance for impurely compiled applications. *fc-userscan* is published as open source software under a 3-clause BSD license. Get the Rust [source code](https://github.com/flyingcircusio/userscan) or the latest [pre-compiled binaries for x86\_64](https://github.com/flyingcircusio/userscan/releases). Pull requests are welcome.

## Example: How to get stale Nix store references

Imagine we have Python installed in our Nix user profile. When creating a Python virtual environment, we get unmanaged references into the Nix store:

```
$ nix-env -iA nixpkgs.python35
$ ~/.nix-profile/bin/pyvenv venv
$ ls -og venv/bin/python3.5
lrwxrwxrwx 1 71 Sep 29 11:19 venv/bin/python3.5 -> /nix/store/8lh4pxhhi4cx00pp1zxpz9pqyy44kjm6-python3-3.5.4/bin/python3.5
```

The garbage collector does not know anything about this reference. This is OK as long as nothing gets deleted from the Nix store. But eventually channel updates arrive. After an seemingly innocent run of *nixos-rebuild* or *nix-env --upgrade*, chances are high that the Python package's hash has changed. The next run of *nix-collect-garbage* will trash the Python binary still referenced from our virtualenv. Although we use Python for illustration, the problem is universal. You can easily run into similar trouble with C programs, for example.

## Protecting unmanaged references

We run *fc-userscan* to detect and register Nix store references:

```
$ fc-userscan -v venv
fc-userscan: Scouting venv
2 references in /nix/var/nix/gcroots/profiles/per-user/ck/home/ck/venv
Processed 741 files (8 MB read) in 0.011 s
fc-userscan: Finished venv
```

Now the exact Python interpreter version is registered as a garbage collection root:

```
$ ls -og /nix/var/nix/gcroots/profiles/per-user/ck/home/ck/venv
lrwxrwxrwx 1 57 Sep 29 11:46 8lh4pxhhi4cx00pp1zxpz9pqyy44kjm6 -> /nix/store/8lh4pxhhi4cx00pp1zxpz9pqyy44kjm6-python3-3.5.4
[...]
```

The Python interpreter and all its dependencies are now protected. The file layout in the *per-user* directories is organized in such a way that is possible to re-scan arbitrary directories and always get correct results.

## Feature overview

### Include/exclude lists

To reduce system load, *fc-userscan* can be instructed to skip files that are unlikely to contain store references. Specify include/exclude globs from

1. a default ignore file **~/.userscan-ignore** if existent (in [gitignore](https://git-scm.com/docs/gitignore)(5) format)
2. a custom ignore file with **-­-exclude-from** (in [gitignore](https://git-scm.com/docs/gitignore)(5) format)
3. the command line with **-­-exclude**, **-­-include**.

A related option is **-­-quickcheck** which causes *fc-userscan* to stop scanning a file when there is no Nix store reference in the first *n* bytes of that file.

### Cache

*fc-userscan* is able to preserve scan results between runs to avoid re-scanning unchanged files. Simply add **-­-cache** *FILE*. It uses the ctime inode attribute to determine if a file has been changed or not. The cache is stored as compressed [messagepack](http://msgpack.org/) file so that it does not take much space even on large installations.

### Unzip on the fly

Some runnable application artefacts are stored as ZIP files and may contain Nix store references. The most prominent instances are compressed Python eggs and JAR files. *fc-userscan* has support to transparently decompress files that match glob patterns passed with **-­-unzip**. Note that scanning inside compressed ZIP archives is significantly slower than scanning regular files.

### List mode

It is possible to scan directory hierarchies and just print all found references to stdout instead of registering them with **-­-list**. The output format is script friendly. These options and many more are explained in the fc-userscan(1) man page which ships with the release.

## Interaction with nix-collect-garbage

How does *fc-userscan* interact with *nix-collect-garbage* now? We at Flying Circus run *fc-userscan* for all users and start *nix-collect-garbage* directly afterwards. *fc-userscan* exits unsuccessfully if there was a problem (e.g., unreadable files). We recommend not to run *nix-collect-garbage* if *fc-userscan* found a problem. Otherwise, not all relevant files might have been checked for Nix store references and *nix-collect-garbage* might delete too much. Here is an example script which scans all users' home directories:

```
failed=0
while read user home; do
  sudo -u $user -H -- \
    fc-userscan -v 2 \
    --exclude-from /etc/userscan.exclude \
    --cache $home/.cache/fc-userscan.cache \
    --unzip '*.egg'
    $home || failed=1
done <