Merge branch 'staging'

<variablelist>

  <varlistentry>

    <term>CC Wrapper</term>

    <term>Bintools Wrapper</term>

    <listitem>

      <para>

        CC Wrapper wraps a C toolchain for a bunch of miscellaneous purposes.

        Specifically, a C compiler (GCC or Clang), Binutils (or the CCTools + binutils mashup when targetting Darwin), and a C standard library (glibc or Darwin's libSystem) are all fed in, and dependency finding, hardening (see below), and purity checks for each are handled by CC Wrapper.

        Packages typically depend on only CC Wrapper, instead of those 3 inputs directly.

        Bintools Wrapper wraps the binary utilities for a bunch of miscellaneous purposes.

        These are GNU Binutils when targetting Linux, and a mix of cctools and GNU binutils for Darwin.

        [The "Bintools" name is supposed to be a compromise between "Binutils" and "cctools" not denoting any specific implementation.]

        Specifically, the underlying bintools package, and a C standard library (glibc or Darwin's libSystem, just for the dynamic loader) are all fed in, and dependency finding, hardening (see below), and purity checks for each are handled by Bintools Wrapper.

        Packages typically depend on CC Wrapper, which in turn (at run time) depends on Bintools Wrapper.

      </para>

      <para>

        Dependency finding is undoubtedly the main task of CC wrapper.

        Bintools Wrapper was only just recently split off from CC Wrapper, so the division of labor is still being worked out.

        For example, it shouldn't care about about the C standard library, but just take a derivation with the dynamic loader (which happens to be the glibc on linux).

        Dependency finding however is a task both wrappers will continue to need to share, and probably the most important to understand.

        It is currently accomplished by collecting directories of host-platform dependencies (i.e. <varname>buildInputs</varname> and <varname>nativeBuildInputs</varname>) in environment variables.

        CC wrapper's setup hook causes any <filename>include</filename> subdirectory of such a dependency to be added to <envar>NIX_CFLAGS_COMPILE</envar>, and any <filename>lib</filename> and <filename>lib64</filename> subdirectories to <envar>NIX_LDFLAGS</envar>.

        The setup hook itself contains some lengthy comments describing the exact convoluted mechanism by which this is accomplished.

        Bintools Wrapper's setup hook causes any <filename>lib</filename> and <filename>lib64</filename> subdirectories to be added to <envar>NIX_LDFLAGS</envar>.

        Since CC Wrapper and Bintools Wrapper use the same strategy, most of the Bintools Wrapper code is sparsely commented and refers to CC Wrapper.

        But CC Wrapper's code, by contrast, has quite lengthy comments.

        Bintools Wrapper merely cites those, rather than repeating them, to avoid falling out of sync.

      </para>

      <para>

        A final task of the setup hook is defining a number of standard environment variables to tell build systems which executables full-fill which purpose.

        They are defined to just be the base name of the tools, under the assumption that CC Wrapper's binaries will be on the path.

        They are defined to just be the base name of the tools, under the assumption that Bintools Wrapper's binaries will be on the path.

        Firstly, this helps poorly-written packages, e.g. ones that look for just <command>gcc</command> when <envar>CC</envar> isn't defined yet <command>clang</command> is to be used.

        Secondly, this helps packages not get confused when cross-compiling, in which case multiple CC wrappers may be simultaneous in use (targeting different platforms).

        <envar>BUILD_</envar>- and <envar>TARGET_</envar>-prefixed versions of the normal environment variable are defined for the additional CC Wrappers, properly disambiguating them.

        Secondly, this helps packages not get confused when cross-compiling, in which case multiple Bintools Wrappers may simultaneously be in use.

        <footnote><para>

          Each wrapper targets a single platform, so if binaries for multiple platforms are needed, the underlying binaries must be wrapped multiple times.

          As this is a property of the wrapper itself, the multiple wrappings are needed whether or not the same underlying binaries can target multiple platforms.

        </para></footnote>

        <envar>BUILD_</envar>- and <envar>TARGET_</envar>-prefixed versions of the normal environment variable are defined for the additional Bintools Wrappers, properly disambiguating them.

      </para>

      <para>

        A problem with this final task is that CC Wrapper is honest and defines <envar>LD</envar> as <command>ld</command>.

        A problem with this final task is that Bintools Wrapper is honest and defines <envar>LD</envar> as <command>ld</command>.

        Most packages, however, firstly use the C compiler for linking, secondly use <envar>LD</envar> anyways, defining it as the C compiler, and thirdly, only so define <envar>LD</envar> when it is undefined as a fallback.

        This triple-threat means CC Wrapper will break those packages, as LD is already defined as the actually linker which the package won't override yet doesn't want to use.

        This triple-threat means Bintools Wrapper will break those packages, as LD is already defined as the actual linker which the package won't override yet doesn't want to use.

        The workaround is to define, just for the problematic package, <envar>LD</envar> as the C compiler.

        A good way to do this would be <command>preConfigure = "LD=$CC"</command>.

      </para>

    </listitem>

  </varlistentry>

  <varlistentry>

    <term>CC Wrapper</term>

    <listitem>

      <para>

        CC Wrapper wraps a C toolchain for a bunch of miscellaneous purposes.

        Specifically, a C compiler (GCC or Clang), wrapped binary tools, and a C standard library (glibc or Darwin's libSystem, just for the dynamic loader) are all fed in, and dependency finding, hardening (see below), and purity checks for each are handled by CC Wrapper.

        Packages typically depend on CC Wrapper, which in turn (at run time) depends on Bintools Wrapper.

      </para>

      <para>

        Dependency finding is undoubtedly the main task of CC Wrapper.

        This works just like Bintools Wrapper, except that any <filename>include</filename> subdirectory of any relevant dependency is added to <envar>NIX_CFLAGS_COMPILE</envar>.

        The setup hook itself contains some lengthy comments describing the exact convoluted mechanism by which this is accomplished.

      </para>

      <para>

        CC Wrapper also like Bintools Wrapper defines standard environment variables with the names of the tools it wraps, for the same reasons described above.

        Importantly, while it includes a <command>cc</command> symlink to the c compiler for portability, the <envar>CC</envar> will be defined using the compiler's "real name" (i.e. <command>gcc</command> or <command>clang</command>).

        This helps lousy build systems that inspect on the name of the compiler rather than run it.

      </para>

    </listitem>

  </varlistentry>

  <varlistentry>

    <term>Perl</term>

    <listitem><para>Adds the <filename>lib/site_perl</filename> subdirectory

{ stdenv, fetchurl, fetchFromGitHub, makeWrapper

{ stdenv, fetchurl, fetchFromGitHub, fetchpatch, makeWrapper

, docutils, perl, pkgconfig, python3, which, ffmpeg

, freefont_ttf, freetype, libass, libpthreadstubs

, lua, lua5_sockets, libuchardet, libiconv ? null, darwin

    sha256 = "0746kmsg69675y5c70vn8imcr9d1zpjz97f27xr1vx00yjpd518v";

  };

  patchPhase = ''

  patches = [

    (fetchpatch {

      url = "https://github.com/mpv-player/mpv/commit/2ecf240b1cd20875991a5b18efafbe799864ff7f.patch";

      sha256 = "1sr0770rvhsgz8d7ysr9qqp4g9gwdhgj8g3rgnz90wl49lgrykhb";

    })

  ];

  postPatch = ''

    patchShebangs ./TOOLS/

  '';

# See cc-wrapper for comments.

var_templates_list=(

    NIX+IGNORE_LD_THROUGH_GCC

    NIX+LDFLAGS

    NIX+LDFLAGS_BEFORE

    NIX+LDFLAGS_AFTER

    NIX+LDFLAGS_HARDEN

)

var_templates_bool=(

    NIX+SET_BUILD_ID

    NIX+DONT_SET_RPATH

)

declare -a role_infixes=()

if [ "${NIX_BINTOOLS_WRAPPER_@infixSalt@_TARGET_BUILD:-}" ]; then

    role_infixes+=(_BUILD_)

fi

if [ "${NIX_BINTOOLS_WRAPPER_@infixSalt@_TARGET_HOST:-}" ]; then

    role_infixes+=(_)

fi

if [ "${NIX_BINTOOLS_WRAPPER_@infixSalt@_TARGET_TARGET:-}" ]; then

    role_infixes+=(_TARGET_)

fi

for var in "${var_templates_list[@]}"; do

    mangleVarList "$var" "${role_infixes[@]}"

done

for var in "${var_templates_bool[@]}"; do

    mangleVarBool "$var" "${role_infixes[@]}"

done

if [ -e @out@/nix-support/libc-ldflags ]; then

    NIX_@infixSalt@_LDFLAGS+=" $(< @out@/nix-support/libc-ldflags)"

fi

if [ -e @out@/nix-support/libc-ldflags-before ]; then

    NIX_@infixSalt@_LDFLAGS_BEFORE="$(< @out@/nix-support/libc-ldflags-before) $NIX_@infixSalt@_LDFLAGS_BEFORE"

fi

export NIX_BINTOOLS_WRAPPER_@infixSalt@_FLAGS_SET=1

hardeningFlags=(relro bindnow)

# Intentionally word-split in case 'hardeningEnable' is defined in

# Nix. Also, our bootstrap tools version of bash is old enough that

# undefined arrays trip `set -u`.

if [[ -v hardeningEnable[@] ]]; then

  hardeningFlags+=(${hardeningEnable[@]})

fi

hardeningLDFlags=()

declare -A hardeningDisableMap

# Intentionally word-split in case 'hardeningDisable' is defined in Nix.

for flag in ${hardeningDisable[@]:-IGNORED_KEY} @hardening_unsupported_flags@

do

  hardeningDisableMap[$flag]=1

done

if (( "${NIX_DEBUG:-0}" >= 1 )); then

  printf 'HARDENING: disabled flags:' >&2

  (( "${#hardeningDisableMap[@]}" )) && printf ' %q' "${!hardeningDisableMap[@]}" >&2

  echo >&2

fi

if [[ -z "${hardeningDisableMap[all]:-}" ]]; then

  if (( "${NIX_DEBUG:-0}" >= 1 )); then

    echo 'HARDENING: Is active (not completely disabled with "all" flag)' >&2;

  fi

  for flag in "${hardeningFlags[@]}"

  do

    if [[ -z "${hardeningDisableMap[$flag]:-}" ]]; then

      case $flag in

        pie)

          if [[ ! ("$*" =~ " -shared " || "$*" =~ " -static ") ]]; then

            if (( "${NIX_DEBUG:-0}" >= 1 )); then echo HARDENING: enabling LDFlags -pie >&2; fi

            hardeningLDFlags+=('-pie')

          fi

          ;;

        relro)

          if (( "${NIX_DEBUG:-0}" >= 1 )); then echo HARDENING: enabling relro >&2; fi

          hardeningLDFlags+=('-z' 'relro')

          ;;

        bindnow)

          if (( "${NIX_DEBUG:-0}" >= 1 )); then echo HARDENING: enabling bindnow >&2; fi

          hardeningLDFlags+=('-z' 'now')

          ;;

        *)

          # Ignore unsupported. Checked in Nix that at least *some*

          # tool supports each flag.

          ;;

      esac

    fi

  done

fi

# The Nixpkgs CC is not directly usable, since it doesn't know where

# the C library and standard header files are. Therefore the compiler

# produced by that package cannot be installed directly in a user

# environment and used from the command line. So we use a wrapper

# script that sets up the right environment variables so that the

# compiler and the linker just "work".

{ name ? "", stdenvNoCC, nativeTools, noLibc ? false, nativeLibc, nativePrefix ? ""

, bintools ? null, libc ? null

, coreutils ? null, shell ? stdenvNoCC.shell, gnugrep ? null

, extraPackages ? [], extraBuildCommands ? ""

, buildPackages ? {}

, useMacosReexportHack ? false

}:

with stdenvNoCC.lib;

assert nativeTools -> nativePrefix != "";

assert !nativeTools ->

  bintools != null && coreutils != null && gnugrep != null;

assert !(nativeLibc && noLibc);

assert (noLibc || nativeLibc) == (libc == null);

let

  stdenv = stdenvNoCC;

  inherit (stdenv) hostPlatform targetPlatform;

  # Prefix for binaries. Customarily ends with a dash separator.

  #

  # TODO(@Ericson2314) Make unconditional, or optional but always true by

  # default.

  targetPrefix = stdenv.lib.optionalString (targetPlatform != hostPlatform)

                                        (targetPlatform.config + "-");

  bintoolsVersion = (builtins.parseDrvName bintools.name).version;

  bintoolsName = (builtins.parseDrvName bintools.name).name;

  libc_bin = if libc == null then null else getBin libc;

  libc_dev = if libc == null then null else getDev libc;

  libc_lib = if libc == null then null else getLib libc;

  bintools_bin = if nativeTools then "" else getBin bintools;

  # The wrapper scripts use 'cat' and 'grep', so we may need coreutils.

  coreutils_bin = if nativeTools then "" else getBin coreutils;

  dashlessTarget = stdenv.lib.replaceStrings ["-"] ["_"] targetPlatform.config;

  # See description in cc-wrapper.

  infixSalt = dashlessTarget;

  # The dynamic linker has different names on different platforms. This is a

  # shell glob that ought to match it.

  dynamicLinker =

    /**/ if libc == null then null

    else if targetPlatform.system == "i686-linux"     then "${libc_lib}/lib/ld-linux.so.2"

    else if targetPlatform.system == "x86_64-linux"   then "${libc_lib}/lib/ld-linux-x86-64.so.2"

    # ARM with a wildcard, which can be "" or "-armhf".

    else if (with targetPlatform; isArm && isLinux)   then "${libc_lib}/lib/ld-linux*.so.3"

    else if targetPlatform.system == "aarch64-linux"  then "${libc_lib}/lib/ld-linux-aarch64.so.1"

    else if targetPlatform.system == "powerpc-linux"  then "${libc_lib}/lib/ld.so.1"

    else if targetPlatform.system == "mips64el-linux" then "${libc_lib}/lib/ld.so.1"

    else if targetPlatform.isDarwin                   then "/usr/lib/dyld"

    else if stdenv.lib.hasSuffix "pc-gnu" targetPlatform.config then "ld.so.1"

    else null;

  expand-response-params =

    if buildPackages.stdenv.cc or null != null && buildPackages.stdenv.cc != "/dev/null"

    then import ../expand-response-params { inherit (buildPackages) stdenv; }

    else "";

in

stdenv.mkDerivation {

  name = targetPrefix

    + (if name != "" then name else "${bintoolsName}-wrapper")

    + (stdenv.lib.optionalString (bintools != null && bintoolsVersion != "") "-${bintoolsVersion}");

  preferLocalBuild = true;

  inherit bintools_bin libc_bin libc_dev libc_lib coreutils_bin;

  shell = getBin shell + shell.shellPath or "";

  gnugrep_bin = if nativeTools then "" else gnugrep;

  inherit targetPrefix infixSalt;

  outputs = [ "out" "info" "man" ];

  passthru = {

    inherit bintools libc nativeTools nativeLibc nativePrefix;

    emacsBufferSetup = pkgs: ''

      ; We should handle propagation here too

      (mapc

        (lambda (arg)

          (when (file-directory-p (concat arg "/lib"))

            (setenv "NIX_${infixSalt}_LDFLAGS" (concat (getenv "NIX_${infixSalt}_LDFLAGS") " -L" arg "/lib")))

          (when (file-directory-p (concat arg "/lib64"))

            (setenv "NIX_${infixSalt}_LDFLAGS" (concat (getenv "NIX_${infixSalt}_LDFLAGS") " -L" arg "/lib64"))))

        '(${concatStringsSep " " (map (pkg: "\"${pkg}\"") pkgs)}))

    '';

  };

  dontBuild = true;

  dontConfigure = true;

  unpackPhase = ''

    src=$PWD

  '';

  installPhase =

    ''

      set -u

      mkdir -p $out/bin {$out,$info,$man}/nix-support

      wrap() {

        local dst="$1"

        local wrapper="$2"

        export prog="$3"

        set +u

        substituteAll "$wrapper" "$out/bin/$dst"

        set -u

        chmod +x "$out/bin/$dst"

      }

    ''

    + (if nativeTools then ''

      echo ${nativePrefix} > $out/nix-support/orig-bintools

      ldPath="${nativePrefix}/bin"

    '' else ''

      echo $bintools_bin > $out/nix-support/orig-bintools

      ldPath="${bintools_bin}/bin"

    ''

    + optionalString (targetPlatform.isSunOS && nativePrefix != "") ''

      # Solaris needs an additional ld wrapper.

      ldPath="${nativePrefix}/bin"

      exec="$ldPath/${targetPrefix}ld"

      wrap ld-solaris ${./ld-solaris-wrapper.sh}

    '')

    + ''

      # Create a symlink to as (the assembler).

      if [ -e $ldPath/${targetPrefix}as ]; then

        ln -s $ldPath/${targetPrefix}as $out/bin/${targetPrefix}as

      fi

    '' + (if !useMacosReexportHack then ''

      wrap ${targetPrefix}ld ${./ld-wrapper.sh} ''${ld:-$ldPath/${targetPrefix}ld}

    '' else ''

      ldInner="${targetPrefix}ld-reexport-delegate"

      wrap "$ldInner" ${./macos-sierra-reexport-hack.bash} ''${ld:-$ldPath/${targetPrefix}ld}

      wrap "${targetPrefix}ld" ${./ld-wrapper.sh} "$out/bin/$ldInner"

      unset ldInner

    '') + ''

      for variant in ld.gold ld.bfd ld.lld; do

        local underlying=$ldPath/${targetPrefix}$variant

        [[ -e "$underlying" ]] || continue

        wrap ${targetPrefix}$variant ${./ld-wrapper.sh} $underlying

      done

      set +u

    '';

  propagatedBuildInputs = extraPackages;

  setupHook = ./setup-hook.sh;

  postFixup =

    ''

      set -u

    ''

    + optionalString (libc != null) (''

      ##

      ## General libc support

      ##

      echo "-L${libc_lib}/lib" > $out/nix-support/libc-ldflags

      echo "${libc_lib}" > $out/nix-support/orig-libc

      echo "${libc_dev}" > $out/nix-support/orig-libc-dev

      ##

      ## Dynamic linker support

      ##

      if [[ -z ''${dynamicLinker+x} ]]; then

        echo "Don't know the name of the dynamic linker for platform '${targetPlatform.config}', so guessing instead." >&2

        local dynamicLinker="${libc_lib}/lib/ld*.so.?"

      fi

      # Expand globs to fill array of options

      dynamicLinker=($dynamicLinker)

      case ''${#dynamicLinker[@]} in

        0) echo "No dynamic linker found for platform '${targetPlatform.config}'." >&2;;

        1) echo "Using dynamic linker: '$dynamicLinker'" >&2;;

        *) echo "Multiple dynamic linkers found for platform '${targetPlatform.config}'." >&2;;

      esac

      if [ -n "''${dynamicLinker:-}" ]; then

        echo $dynamicLinker > $out/nix-support/dynamic-linker

    '' + (if targetPlatform.isDarwin then ''

        printf "export LD_DYLD_PATH=%q\n" "$dynamicLinker" >> $out/nix-support/setup-hook

    '' else ''

        if [ -e ${libc_lib}/lib/32/ld-linux.so.2 ]; then

          echo ${libc_lib}/lib/32/ld-linux.so.2 > $out/nix-support/dynamic-linker-m32

        fi

        local ldflagsBefore=(-dynamic-linker "$dynamicLinker")

    '') + ''

      fi

      # The dynamic linker is passed in `ldflagsBefore' to allow

      # explicit overrides of the dynamic linker by callers to ld

      # (the *last* value counts, so ours should come first).

      printWords "''${ldflagsBefore[@]}" > $out/nix-support/libc-ldflags-before

    '')

    + optionalString (!nativeTools) ''

      ##

      ## User env support

      ##

      # Propagate the underling unwrapped bintools so that if you

      # install the wrapper, you get tools like objdump, the manpages,

      # etc. as well (same for any binaries of libc).

      printWords ${bintools_bin} ${if libc == null then "" else libc_bin} > $out/nix-support/propagated-user-env-packages

      ##

      ## Man page and info support

      ##

      printWords ${bintools.info or ""} \

        >> $info/nix-support/propagated-build-inputs

      printWords ${bintools.man or ""} \

        >> $man/nix-support/propagated-build-inputs

    ''

    + ''

      ##

      ## Hardening support

      ##

      # some linkers on some platforms don't support specific -z flags

      export hardening_unsupported_flags=""

      if [[ "$($ldPath/${targetPrefix}ld -z now 2>&1 || true)" =~ un(recognized|known)\ option ]]; then

        hardening_unsupported_flags+=" bindnow"

      fi

      if [[ "$($ldPath/${targetPrefix}ld -z relro 2>&1 || true)" =~ un(recognized|known)\ option ]]; then

        hardening_unsupported_flags+=" relro"

      fi

    ''

    + optionalString hostPlatform.isCygwin ''

      hardening_unsupported_flags+=" pic"

    ''

    + ''

      set +u

      substituteAll ${./add-flags.sh} $out/nix-support/add-flags.sh

      substituteAll ${./add-hardening.sh} $out/nix-support/add-hardening.sh

      substituteAll ${../cc-wrapper/utils.sh} $out/nix-support/utils.sh

      ##

      ## Extra custom steps

      ##

    ''

    + extraBuildCommands;

  inherit dynamicLinker expand-response-params;

  # for substitution in utils.sh

  expandResponseParams = "${expand-response-params}/bin/expand-response-params";

  meta =

    let bintools_ = if bintools != null then bintools else {}; in

    (if bintools_ ? meta then removeAttrs bintools.meta ["priority"] else {}) //

    { description =

        stdenv.lib.attrByPath ["meta" "description"] "System binary utilities" bintools_

        + " (wrapper script)";

  } // optionalAttrs useMacosReexportHack {

    platforms = stdenv.lib.platforms.darwin;

  };

}