ch05-repository-admin.xml - svnbook - Project Hosting on Google Code

‹r3305

r3707

Source path: svn/ tags/ en-1.5-final/ src/ en/ book/ ch05-repository-admin.xml

<chapter id="svn.reposadmin">
  <title>Repository Administration</title>

  <para>The Subversion repository is the central storehouse of all
    your versioned data.  As such, it becomes an obvious candidate for
    all the love and attention an administrator can offer.  While the
    repository is generally a low-maintenance item, it is important to
    understand how to properly configure and care for it so that
    potential problems are avoided, and so actual problems are safely
    resolved.</para>

  <para>In this chapter, we'll discuss how to create and configure a
    Subversion repository.  We'll also talk about repository
    maintenance, providing examples of how and when to use the
    <command>svnlook</command> and <command>svnadmin</command> tools
    provided with Subversion.  We'll address some common questions and
    mistakes and give some suggestions on how to arrange the data in
    the repository.</para>

  <para>If you plan to access a Subversion repository only in the
    role of a user whose data is under version control (i.e., via
    a Subversion client), you can skip this chapter altogether.
    However, if you are, or wish to become, a Subversion repository
    administrator,
    <footnote>
      <para>This may sound really prestigious and lofty, but we're
        just talking about anyone who is interested in that
        mysterious realm beyond the working copy where everyone's
        data hangs out.</para>
    </footnote>
    this chapter is for you.</para>


  <!-- ================================================================= -->
  <!-- ================================================================= -->
  <!-- ================================================================= -->
  <sect1 id="svn.reposadmin.basics">
    <title>The Subversion Repository, Defined</title>

    <para>Before jumping into the broader topic of repository
      administration, let's further define what a repository is.  How
      does it look?  How does it feel?  Does it take its tea hot or
      iced, sweetened, and with lemon?  As an administrator, you'll be
      expected to understand the composition of a repository both from
      a literal, OS-level perspective&mdash;how a repository looks and
      acts with respect to non-Subversion tools&mdash;and from a
      logical perspective&mdash;dealing with how data is represented
      <emphasis>inside</emphasis> the repository.</para>

    <para>Seen through the eyes of a typical file browser application
      (such as Windows Explorer) or command-line based filesystem
      navigation tools, the Subversion repository is just another
      directory full of stuff.  There are some subdirectories with
      human-readable configuration files in them, some subdirectories
      with some not-so-human-readable data files, and so on.  As in
      other areas of the Subversion design, modularity is given high
      regard, and hierarchical organization is preferred to cluttered
      chaos.  So a shallow glance into a typical repository from a
      nuts-and-bolts perspective is sufficient to reveal the basic
      components of the repository:</para>
            
    <screen>
$ ls repos
conf/  dav/  db/  format  hooks/  locks/  README.txt
</screen>

    <para>Here's a quick fly-by overview of what exactly you're seeing
      in this directory listing.  (Don't get bogged down in the
      terminology&mdash;detailed coverage of these components exists
      elsewhere in this and other chapters.)</para>

    <variablelist>
      <varlistentry>
        <term>conf</term>
        <listitem>
          <para>A directory containing configuration files</para>
        </listitem>
      </varlistentry>
      <varlistentry>
        <term>dav</term>
        <listitem>
          <para>A directory provided to
            <filename>mod_dav_svn</filename> for its private
            housekeeping data</para>
        </listitem>
      </varlistentry>
      <varlistentry>
        <term>db</term>
        <listitem>
          <para>The data store for all of your versioned data</para>
        </listitem>
      </varlistentry>
      <varlistentry>
        <term>format</term>
        <listitem>
          <para>A file that contains a single integer that
            indicates the version number of the repository layout</para>
        </listitem>
      </varlistentry>
      <varlistentry>
        <term>hooks</term>
        <listitem>
          <para>A directory full of hook script templates (and hook
            scripts themselves, once you've installed some)</para>
        </listitem>
      </varlistentry>
      <varlistentry>
        <term>locks</term>
        <listitem>
          <para>A directory for Subversion's repository lock
            files, used for tracking accessors to the repository</para>
        </listitem>
      </varlistentry>
      <varlistentry>
        <term>README.txt</term>
        <listitem>
          <para>A file whose contents merely inform its readers that
            they are looking at a Subversion repository</para>
        </listitem>
      </varlistentry>
    </variablelist>

    <para>Of course, when accessed via the Subversion libraries, this
      otherwise unremarkable collection of files and directories
      suddenly becomes an implementation of a virtual, versioned
      filesystem, complete with customizable event triggers.  This
      filesystem has its own notions of directories and files, very
      similar to the notions of such things held by real filesystems
      (such as NTFS, FAT32, ext3, etc.).  But this is a special
      filesystem&mdash;it hangs these directories and files from
      revisions, keeping all the changes you've ever made to them
      safely stored and forever accessible.  This is where the
      entirety of your versioned data lives.</para>

  </sect1>
 
  <!-- ================================================================= -->
  <!-- ================================================================= -->
  <!-- ================================================================= -->
  <sect1 id="svn.reposadmin.planning">
    <title>Strategies for Repository Deployment</title>

    <para>Due largely to the simplicity of the overall design of the
      Subversion repository and the technologies on which it relies,
      creating and configuring a repository are fairly straightforward
      tasks.  There are a few preliminary decisions you'll want to
      make, but the actual work involved in any given setup of a
      Subversion repository is pretty basic, tending toward
      mindless repetition if you find yourself setting up multiples of
      these things.</para>

    <para>Some things you'll want to consider beforehand, though, are:</para>

    <itemizedlist>
      <listitem>
        <para>What data do you expect to live in your repository (or
          repositories), and how will that data be organized?</para>
      </listitem>
      <listitem>
        <para>Where will your repository live, and how will it be
          accessed?</para>
      </listitem>
      <listitem>
        <para>What types of access control and repository event
          reporting do you need?</para>
      </listitem>
      <listitem>
        <para>Which of the available types of data store do you want
          to use?</para>
      </listitem>
    </itemizedlist>

    <para>In this section, we'll try to help you answer those
      questions.</para>

    <!-- =============================================================== -->
    <sect2 id="svn.reposadmin.projects.chooselayout">
      <title>Planning Your Repository Organization</title>

      <para>While Subversion allows you to move around versioned files
        and directories without any loss of information, and even
        provides ways of moving whole sets of versioned history from
        one repository to another, doing so can greatly disrupt the
        workflow of those who access the repository often and come to
        expect things to be at certain locations.  So before creating
        a new repository, try to peer into the future a bit; plan
        ahead before placing your data under version control.  By
        conscientiously <quote>laying out</quote> your repository or
        repositories and their versioned contents ahead of time, you
        can prevent many future headaches.</para>

      <para>Let's assume that as repository administrator, you will be
        responsible for supporting the version control system for
        several projects.  Your first decision is whether to use a
        single repository for multiple projects, or to give each
        project its own repository, or some compromise of these
        two.</para>

      <para>There are benefits to using a single repository for
        multiple projects, most obviously the lack of duplicated
        maintenance.  A single repository means that there is one set
        of hook programs, one thing to routinely back up, one thing to
        dump and load if Subversion releases an incompatible new
        version, and so on.  Also, you can move data between projects
        easily, without losing any historical versioning
        information.</para>


      <para>The downside of using a single repository is that
        different projects may have different requirements in terms of
        the repository event triggers, such as needing to send commit
        notification emails to different mailing lists, or having
        different definitions about what does and does not constitute
        a legitimate commit.  These aren't insurmountable problems, of
        course&mdash;it just means that all of your hook scripts have
        to be sensitive to the layout of your repository rather than
        assuming that the whole repository is associated with a single
        group of people.  Also, remember that Subversion uses
        repository-global revision numbers.  While those numbers don't
        have any particular magical powers, some folks still don't
        like the fact that even though no changes have been made to
        their project lately, the youngest revision number for the
        repository keeps climbing because other projects are actively
        adding new revisions.
        <footnote>
          <para>Whether founded in ignorance or in poorly considered
            concepts about how to derive legitimate software
            development metrics, global revision numbers are a silly
            thing to fear, and <emphasis>not</emphasis> the kind of
            thing you should weigh when deciding how to arrange your
            projects and repositories.</para>
        </footnote>
      </para>

      <para>A middle-ground approach can be taken, too.  For example,
        projects can be grouped by how well they relate to each other.
        You might have a few repositories with a handful of projects
        in each repository.  That way, projects that are likely to
        want to share data can do so easily, and as new revisions are
        added to the repository, at least the developers know that
        those new revisions are at least remotely related to everyone
        who uses that repository.</para>

      <para>After deciding how to organize your projects with respect
        to repositories, you'll probably want to think about directory
        hierarchies within the repositories themselves.  Because
        Subversion uses regular directory copies for branching and
        tagging (see <xref linkend="svn.branchmerge"/>), the
        Subversion community recommends that you choose a repository
        location for each <firstterm>project
        root</firstterm>&mdash;the <quote>topmost</quote> directory
        that contains data related to that project&mdash;and then
        create three subdirectories beneath that root:
        <filename>trunk</filename>, meaning the directory under which
        the main project development occurs;
        <filename>branches</filename>, which is a directory in which
        to create various named branches of the main development line;
        and <filename>tags</filename>, which is a collection of tree
        snapshots that are created, and perhaps destroyed, but never
        changed.
        <footnote>
          <para>The <filename>trunk</filename>, <filename>tags</filename>, 
            and <filename>branches</filename> trio is sometimes referred
            to as <quote>the TTB directories.</quote></para>
        </footnote>
        </para>

      <para>For example, your repository might look like this:</para>

      <screen>
/
   calc/
      trunk/
      tags/
      branches/
   calendar/
      trunk/
      tags/
      branches/
   spreadsheet/
      trunk/
      tags/
      branches/
   &hellip;
</screen>

      <para>Note that it doesn't matter where in your repository each
        project root is.  If you have only one project per repository,
        the logical place to put each project root is at the root of
        that project's respective repository.  If you have multiple
        projects, you might want to arrange them in groups inside the
        repository, perhaps putting projects with similar goals or
        shared code in the same subdirectory, or maybe just grouping
        them alphabetically.  Such an arrangement might look
        like this:</para>

      <screen>
/
   utils/
      calc/
         trunk/
         tags/
         branches/
      calendar/
         trunk/
         tags/
         branches/
      &hellip;
   office/
      spreadsheet/
         trunk/
         tags/
         branches/
      &hellip;
</screen>

      <para>Lay out your repository in whatever way you see fit.
        Subversion does not expect or enforce a particular layout&mdash;in
        its eyes, a directory is a directory is a directory.
        Ultimately, you should choose the repository arrangement that
        meets the needs of the people who work on the projects that
        live there.</para>

      <para>In the name of full disclosure, though, we'll mention
        another very common layout.  In this layout, the
        <filename>trunk</filename>, <filename>tags</filename>, and
        <filename>branches</filename> directories live in the root
        directory of your repository, and your projects are in
        subdirectories beneath those, like so:</para>

      <screen>
/
   trunk/
      calc/
      calendar/
      spreadsheet/
      &hellip;
   tags/
      calc/
      calendar/
      spreadsheet/
      &hellip;
   branches/
      calc/
      calendar/
      spreadsheet/
      &hellip;
</screen>

      <para>There's nothing particularly incorrect about such a
        layout, but it may or may not seem as intuitive for your
        users.  Especially in large, multiproject situations with
        many users, those users may tend to be familiar with only one
        or two of the projects in the repository.  But the
        projects-as-branch-siblings approach tends to deemphasize project
        individuality and focus on the entire set of projects as a
        single entity.  That's a social issue, though.  We like our
        originally suggested arrangement for purely practical
        reasons&mdash;it's easier to ask about (or modify, or migrate
        elsewhere) the entire history of a single project when there's
        a single repository path that holds the entire
        history&mdash;past, present, tagged, and branched&mdash;for
        that project and that project alone.</para>

    </sect2>

    <!-- =============================================================== -->
    <sect2 id="svn.reposadmin.basics.hosting">
      <title>Deciding Where and How to Host Your Repository</title>

      <para>Before creating your Subversion repository, an obvious
        question you'll need to answer is where the thing is going to
        live.  This is strongly connected to myriad other
        questions involving how the repository will be accessed (via a
        Subversion server or directly), by whom (users behind your
        corporate firewall or the whole world out on the open
        Internet), what other services you'll be providing around
        Subversion (repository browsing interfaces, email-based
        commit notification, etc.), your data backup strategy, and so
        on.</para>

      <para>We cover server choice and configuration in <xref
        linkend="svn.serverconfig" />, but the point we'd like to
        briefly make here is simply that the answers to some of these
        other questions might have implications that force your hand
        when deciding where your repository will live.  For example,
        certain deployment scenarios might require accessing the
        repository via a remote filesystem from multiple computers, in
        which case (as you'll read in the next section) your choice of
        a repository backend data store turns out not to be a choice
        at all because only one of the available backends will work
        in this scenario.</para>

      <para>Addressing each possible way to deploy
        Subversion is both impossible and outside the scope of this
        book.  We simply encourage you to evaluate your options using
        these pages and other sources as your reference material and to
        plan ahead.</para>

    </sect2>

    <!-- =============================================================== -->
    <sect2 id="svn.reposadmin.basics.backends">
      <title>Choosing a Data Store</title>

      <para>As of version 1.1, Subversion provides two options for the
        type of underlying data store&mdash;often referred to as
        <quote>the backend</quote> or, somewhat confusingly,
        <quote>the (versioned) filesystem</quote>&mdash;that each
        repository uses.  One type of data store keeps everything in a
        Berkeley DB (or BDB) database environment; repositories that
        use this type are often referred to as being
        <quote>BDB-backed.</quote>  The other type stores data in
        ordinary flat files, using a custom format.  Subversion
        developers have adopted the habit of referring to this latter
        data storage mechanism as <firstterm>FSFS</firstterm>
        <footnote>
          <para>Often pronounced <quote>fuzz-fuzz,</quote> if Jack
            Repenning has anything to say about it.  (This book,
            however, assumes that the reader is thinking
            <quote>eff-ess-eff-ess.</quote>)</para>
        </footnote> 
        &mdash;a versioned filesystem implementation that uses the
        native OS filesystem directly&mdash;rather than via a database
        library or some other abstraction layer&mdash;to store data.</para>


      <para><xref linkend="svn.reposadmin.basics.backends.tbl-1" />
        gives a comparative overview of Berkeley DB and FSFS
        repositories.</para>

      <table id="svn.reposadmin.basics.backends.tbl-1">
        <title>Repository data store comparison</title>
        <tgroup cols="4">
          <thead>
            <row>
              <entry>Category</entry>
              <entry>Feature</entry>
              <entry>Berkeley DB</entry>
              <entry>FSFS</entry>
            </row>
          </thead>
          <tbody>
            <row>
              <entry morerows="1">Reliability</entry>
              <entry>Data integrity</entry>
              <entry>When properly deployed, extremely reliable;
                Berkeley DB 4.4 brings auto-recovery</entry>
              <entry>Older versions had some rarely demonstrated, but
                data-destroying bugs</entry>
            </row>
            <row>
              <entry>Sensitivity to interruptions</entry>
              <entry>Very; crashes and permission problems can leave the
                database <quote>wedged,</quote> requiring journaled
                recovery procedures</entry>
              <entry>Quite insensitive</entry>
            </row>
            <row>
              <entry morerows="3">Accessibility</entry>
              <entry>Usable from a read-only mount</entry>
              <entry>No</entry>
              <entry>Yes</entry>
            </row>
            <row>
              <entry>Platform-independent storage</entry>
              <entry>No</entry>
              <entry>Yes</entry>
            </row>
            <row>
              <entry>Usable over network filesystems</entry>
              <entry>Generally, no</entry>
              <entry>Yes</entry>
            </row>
            <row>
              <entry>Group permissions handling</entry>
              <entry>Sensitive to user umask problems; best if accessed
                by only one user</entry>
              <entry>Works around umask problems</entry>
            </row>
            <row>
              <entry morerows="2">Scalability</entry>
              <entry>Repository disk usage</entry>
              <entry>Larger (especially if logfiles aren't purged)</entry>
              <entry>Smaller</entry>
            </row>
            <row>
              <entry>Number of revision trees</entry>
              <entry>Database; no problems</entry>
              <entry>Some older native filesystems don't scale well with
                thousands of entries in a single directory</entry>
            </row>
            <row>
              <entry>Directories with many files</entry>
              <entry>Slower</entry>
              <entry>Faster</entry>
            </row>
            <row>
              <entry morerows="1">Performance</entry>
              <entry>Checking out latest revision</entry>
              <entry>No meaningful difference</entry>
              <entry>No meaningful difference</entry>
            </row>
            <row>
              <entry>Large commits</entry>
              <entry>Slower overall, but cost is amortized across the
                lifetime of the commit</entry>
              <entry>Faster overall, but finalization delay may cause 
                client timeouts</entry>
            </row>
          </tbody>
        </tgroup>      
      </table>

      <para>There are advantages and disadvantages to each of these
        two backend types.  Neither of them is more
        <quote>official</quote> than the other, though the newer FSFS
        is the default data store as of Subversion 1.2.  Both are
        reliable enough to trust with your versioned data.  But as you
        can see in <xref
        linkend="svn.reposadmin.basics.backends.tbl-1" />, the FSFS
        backend provides quite a bit more flexibility in terms of its
        supported deployment scenarios.  More flexibility means you
        have to work a little harder to find ways to deploy it
        incorrectly.  Those reasons&mdash;plus the fact that not using
        Berkeley DB means there's one fewer component in the
        system&mdash;largely explain why today almost everyone uses
        the FSFS backend when creating new repositories.</para>

      <para>Fortunately, most programs that access Subversion
        repositories are blissfully ignorant of which backend data
        store is in use.  And you aren't even necessarily stuck with
        your first choice of a data store&mdash;in the event that you
        change your mind later, Subversion provides ways of migrating
        your repository's data into another repository that uses a
        different backend data store.  We talk more about that later
        in this chapter.</para>

      <para>The following subsections provide a more detailed look at
        the available backend data store types.</para>

      <!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -->
      <sect3 id="svn.reposadmin.basics.backends.bdb">
        <title>Berkeley DB</title>
        
        <para>When the initial design phase of Subversion was in
          progress, the developers decided to use Berkeley DB for a
          variety of reasons, including its open source license,
          transaction support, reliability, performance, API
          simplicity, thread safety, support for cursors, and so
          on.</para>

        <para>Berkeley DB provides real transaction
          support&mdash;perhaps its most powerful feature.  Multiple
          processes accessing your Subversion repositories don't have
          to worry about accidentally clobbering each other's data.
          The isolation provided by the transaction system is such
          that for any given operation, the Subversion repository code
          sees a static view of the database&mdash;not a database that
          is constantly changing at the hand of some other
          process&mdash;and can make decisions based on that view.  If
          the decision made happens to conflict with what another
          process is doing, the entire operation is rolled back as though
          it never happened, and Subversion gracefully retries the
          operation against a new, updated (and yet still static) view
          of the database.</para>

        <para>Another great feature of Berkeley DB is <firstterm>hot
          backups</firstterm>&mdash;the ability to back up the
          database environment without taking it
          <quote>offline.</quote> We'll discuss how to back up your
          repository later in this chapter (in <xref
          linkend="svn.reposadmin.maint.backup"/>), but the benefits
          of being able to make fully functional copies of your
          repositories without any downtime should be obvious.</para>

        <para>Berkeley DB is also a very reliable database system when
          properly used.  Subversion uses Berkeley DB's logging
          facilities, which means that the database first writes to
          on-disk logfiles a description of any modifications it is
          about to make, and then makes the modification itself.  This
          is to ensure that if anything goes wrong, the database
          system can back up to a previous
          <firstterm>checkpoint</firstterm>&mdash;a location in the
          logfiles known not to be corrupt&mdash;and replay
          transactions until the data is restored to a usable state.
          See <xref linkend="svn.reposadmin.maint.diskspace"/> later
          in this chapter for more about Berkeley DB logfiles.</para>

        <para>But every rose has its thorn, and so we must note some
          known limitations of Berkeley DB.  First, Berkeley DB
          environments are not portable.  You cannot simply copy a
          Subversion repository that was created on a Unix system onto
          a Windows system and expect it to work.  While much of the
          Berkeley DB database format is architecture-independent,
          other aspects of the environment are not.
          Second, Subversion uses Berkeley DB in a way that will not
          operate on Windows 95/98 systems&mdash;if you need to house
          a BDB-backed repository on a Windows machine, stick with
          Windows 2000 or later.</para>

        <para>While Berkeley DB promises to behave correctly on
          network shares that meet a particular set of specifications,
          <footnote>
            <para>Berkeley DB requires that the underlying filesystem
              implement strict POSIX locking semantics, and more
              importantly, the ability to map files directly into
              process memory.</para>
          </footnote>
          most networked filesystem types and appliances do
          <emphasis>not</emphasis> actually meet those requirements.
          And in no case can you allow a BDB-backed repository that
          resides on a network share to be accessed by multiple
          clients of that share at once (which quite often is the
          whole point of having the repository live on a network share
          in the first place).</para>

        <warning>
          <para>If you attempt to use Berkeley DB on a noncompliant
            remote filesystem, the results are unpredictable&mdash;you
            may see mysterious errors right away, or it may be months
            before you discover that your repository database is
            subtly corrupted.  You should strongly consider using the
            FSFS data store for repositories that need to live on a
            network share.</para>
        </warning>
          
        <para>Finally, because Berkeley DB is a library linked
          directly into Subversion, it's more sensitive to
          interruptions than a typical relational database system.
          Most SQL systems, for example, have a dedicated server
          process that mediates all access to tables.  If a program
          accessing the database crashes for some reason, the database
          daemon notices the lost connection and cleans up any mess
          left behind.  And because the database daemon is the only
          process accessing the tables, applications don't need to
          worry about permission conflicts.  These things are not the
          case with Berkeley DB, however.  Subversion (and programs
          using Subversion libraries) access the database tables
          directly, which means that a program crash can leave the
          database in a temporarily inconsistent, inaccessible state.
          When this happens, an administrator needs to ask Berkeley DB
          to restore to a checkpoint, which is a bit of an annoyance.
          Other things can cause a repository to <quote>wedge</quote>
          besides crashed processes, such as programs conflicting over
          ownership and permissions on the database files.</para>


        <note>
          <para>Berkeley DB 4.4 brings (to Subversion 1.4 and later)
            the ability for Subversion to automatically and
            transparently recover Berkeley DB environments in need of
            such recovery.  When a Subversion process attaches to a
            repository's Berkeley DB environment, it uses some process
            accounting mechanisms to detect any unclean disconnections
            by previous processes, performs any necessary recovery,
            and then continues on as though nothing happened.  This
            doesn't completely eliminate instances of repository
            wedging, but it does drastically reduce the amount of
            human interaction required to recover from them.</para>
        </note>

        <para>So while a Berkeley DB repository is quite fast and
          scalable, it's best used by a single server process running
          as one user&mdash;such as Apache's <command>httpd</command>
          or <command>svnserve</command> (see <xref
          linkend="svn.serverconfig"/>)&mdash;rather than accessing it
          as many different users via <literal>file://</literal> or
          <literal>svn+ssh://</literal> URLs.  If you're accessing a Berkeley
          DB repository directly as multiple users, be sure to read
          <xref linkend="svn.serverconfig.multimethod"/> later in this
          chapter.</para>

      </sect3>
      
      <!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -->
      <sect3 id="svn.reposadmin.basics.backends.fsfs">
        <title>FSFS</title>

        <para>In mid-2004, a second type of repository storage
          system&mdash;one that doesn't use a database at
          all&mdash;came into being.  An FSFS repository stores the
          changes associated with a revision in a single file, and so
          all of a repository's revisions can be found in a single
          subdirectory full of numbered files.  Transactions are
          created in separate subdirectories as individual files.
          When complete, the transaction file is renamed and moved
          into the revisions directory, thus guaranteeing that commits
          are atomic.  And because a revision file is permanent and
          unchanging, the repository also can be backed up while
          <quote>hot,</quote> just like a BDB-backed
          repository.</para>

        <para>The FSFS revision files describe a revision's
          directory structure, file contents, and deltas against files
          in other revision trees.  Unlike a Berkeley DB database,
          this storage format is portable across different operating
          systems and isn't sensitive to CPU architecture.  Because
          no journaling or shared-memory files are being used, the
          repository can be safely accessed over a network filesystem
          and examined in a read-only environment.  The lack of
          database overhead also means the overall repository
          size is a bit smaller.</para>

        <para>FSFS has different performance characteristics, too.
          When committing a directory with a huge number of files,
          FSFS is able to more quickly append directory entries.  On
          the other hand, FSFS writes the latest version of a file as
          a delta against an earlier version, which means that
          checking out the latest tree is a bit slower than fetching
          the full-texts stored in a Berkeley DB HEAD revision.  FSFS
          also has a longer delay when finalizing a commit, which
          could in extreme cases cause clients to time out while
          waiting for a response.</para>

        <para>The most important distinction, however, is FSFS's
          imperviousness to wedging when something goes wrong.  If a
          process using a Berkeley DB database runs into a permissions
          problem or suddenly crashes, the database can be left in an
          unusable state until an administrator recovers it.  If the
          same scenarios happen to a process using an FSFS repository,
          the repository isn't affected at all.  At worst, some
          transaction data is left behind.</para>

        <para>The only real argument against FSFS is its relative
          immaturity compared to Berkeley DB.  Unlike Berkeley DB,
          which has years of history, its own dedicated development
          team, and, now, Oracle's mighty name attached to it,
          <footnote>
            <para>Oracle bought Sleepycat and its flagship software,
              Berkeley DB, on Valentine's Day in 2006.</para>
          </footnote>
          FSFS is a newer bit of engineering.  Prior to Subversion
          1.4, it was still shaking out some pretty serious data
          integrity bugs, which, while triggered in only very rare
          cases, nonetheless did occur.  That said, FSFS has quickly
          become the backend of choice for some of the largest public
          and private Subversion repositories, and it promises a lower
          barrier to entry for Subversion across the board.</para>

      </sect3>
    </sect2>

  </sect1>
 
  <!-- ================================================================= -->
  <!-- ================================================================= -->
  <!-- ================================================================= -->
  <sect1 id="svn.reposadmin.create">
    <title>Creating and Configuring Your Repository</title>

    <para>Earlier in this chapter (in <xref linkend="svn.reposadmin.planning" />), we
      looked at some of the important decisions that should be made
      before creating and configuring your Subversion repository.
      Now, we finally get to get our hands dirty!  In this section,
      we'll see how to actually create a Subversion repository and
      configure it to perform custom actions when special repository
      events occur.</para>

    <!-- =============================================================== -->
    <sect2 id="svn.reposadmin.basics.creating">
      <title>Creating the Repository</title>
   
      <para>Subversion repository creation is an incredibly simple
        task.  The <command>svnadmin</command> utility that comes with
        Subversion provides a subcommand (<command>svnadmin
        create</command>) for doing just that.</para>
          
      <screen>
$ # Create a repository
$ svnadmin create /var/svn/repos
$
</screen>
          
      <para>This creates a new repository in the directory
        <filename>/var/svn/repos</filename>, and with the default
        filesystem data store.  Prior to Subversion 1.2, the default
        was to use Berkeley DB; the default is now FSFS.  You can
        explicitly choose the filesystem type using the
        <option>--fs-type</option> argument, which accepts as a
        parameter either <literal>fsfs</literal> or
        <literal>bdb</literal>.</para>
 
      <screen>
$ # Create an FSFS-backed repository
$ svnadmin create --fs-type fsfs /var/svn/repos
$
</screen>

      <screen>
# Create a Berkeley-DB-backed repository
$ svnadmin create --fs-type bdb /var/svn/repos
$
</screen>
              
      <para>After running this simple command, you have a Subversion
        repository.</para>

      <tip>
        <para>The path argument to <command>svnadmin</command> is just
          a regular filesystem path and not a URL like the
          <command>svn</command> client program uses when referring to
          repositories.  Both <command>svnadmin</command> and
          <command>svnlook</command> are considered server-side
          utilities&mdash;they are used on the machine where the
          repository resides to examine or modify aspects of the
          repository, and are in fact unable to perform tasks across a
          network.  A common mistake made by Subversion newcomers is
          trying to pass URLs (even <quote>local</quote>
          <literal>file://</literal> ones) to these two programs.</para>
      </tip>

      <para>Present in the <filename>db/</filename> subdirectory of
        your repository is the implementation of the versioned
        filesystem. Your new repository's versioned filesystem begins
        life at revision 0, which is defined to consist of nothing but
        the top-level root (<filename>/</filename>) directory.
        Initially, revision 0 also has a single revision property,
        <literal>svn:date</literal>, set to the time at which the
        repository was created.</para>

      <para>Now that you have a repository, it's time to customize
        it.</para>

      <warning>
        <para>While some parts of a Subversion repository&mdash;such
          as the configuration files and hook scripts&mdash;are meant
          to be examined and modified manually, you shouldn't (and
          shouldn't need to) tamper with the other parts of the
          repository <quote>by hand.</quote>  The
          <command>svnadmin</command> tool should be sufficient for
          any changes necessary to your repository, or you can look to
          third-party tools (such as Berkeley DB's tool suite) for
          tweaking relevant subsections of the repository.  Do
          <emphasis>not</emphasis> attempt manual manipulation of your
          version control history by poking and prodding around in
          your repository's data store files!</para>
      </warning>

    </sect2>

    <!-- =============================================================== -->
    <sect2 id="svn.reposadmin.create.hooks">
      <title>Implementing Repository Hooks</title>

      <para>A <firstterm>hook</firstterm> is a program triggered by
        some repository event, such as the creation of a new revision
        or the modification of an unversioned property.  Some hooks
        (the so-called <quote>pre hooks</quote>) run in advance of a
        repository operation and provide a means by which to both
        report what is about to happen and prevent it from
        happening at all.  Other hooks (the <quote>post hooks</quote>)
        run after the completion of a repository event and are useful
        for performing tasks that examine&mdash;but don't
        modify&mdash;the repository.  Each hook is handed enough
        information to tell what that event is (or was), the specific
        repository changes proposed (or completed), and the username
        of the person who triggered the event.</para>
            
      <para>The <filename>hooks</filename> subdirectory is, by
        default, filled with templates for various repository
        hooks:</para>
            
      <screen>
$ ls repos/hooks/
post-commit.tmpl          post-unlock.tmpl  pre-revprop-change.tmpl
post-lock.tmpl            pre-commit.tmpl   pre-unlock.tmpl
post-revprop-change.tmpl  pre-lock.tmpl     start-commit.tmpl
$
</screen>
            
      <para>There is one template for each hook that the Subversion
        repository supports; by examining the contents of those
        template scripts, you can see what triggers each script
        to run and what data is passed to that script.  Also present
        in many of these templates are examples of how one might use
        that script, in conjunction with other Subversion-supplied
        programs, to perform common useful tasks.  To actually install
        a working hook, you need only place some executable program or
        script into the <filename>repos/hooks</filename> directory,
        which can be executed as the name (such as
        <command>start-commit</command> or
        <command>post-commit</command>) of the hook.</para>


      <para>On Unix platforms, this means supplying a script or
        program (which could be a shell script, a Python program, a
        compiled C binary, or any number of other things) named
        exactly like the name of the hook.  Of course, the template
        files are present for more than just informational
        purposes&mdash;the easiest way to install a hook on Unix
        platforms is to simply copy the appropriate template file to a
        new file that lacks the <filename>.tmpl</filename> extension,
        customize the hook's contents, and ensure that the script is
        executable.  Windows, however, uses file extensions to
        determine whether a program is executable, so you would
        need to supply a program whose basename is the name of the
        hook and whose extension is one of the special extensions
        recognized by Windows for executable programs, such as
        <filename>.exe</filename> for programs and
        <filename>.bat</filename> for batch files.</para>

      <tip>
        <para>For security reasons, the Subversion repository executes
          hook programs with an empty environment&mdash;that is, no
          environment variables are set at all, not even
          <literal>$PATH</literal> (or <literal>%PATH%</literal>,
          under Windows).  Because of this, many administrators
          are baffled when their hook program runs fine by hand, but
          doesn't work when run by Subversion.  Be sure to explicitly
          set any necessary environment variables in your hook program
          and/or use absolute paths to programs.</para>
      </tip>

      <para>Subversion executes hooks as the same user who owns the
        process that is accessing the Subversion repository.  In most
        cases, the repository is being accessed via a Subversion
        server, so this user is the same user as whom the server
        runs on the system.  The hooks themselves will need to be
        configured with OS-level permissions that allow that user to
        execute them.  Also, this means that any programs or files
        (including the Subversion repository) accessed directly
        or indirectly by the hook will be accessed as the same user.
        In other words, be alert to potential permission-related
        problems that could prevent the hook from performing the tasks
        it is designed to perform.</para>

      <para>There are serveral hooks implemented by the Subversion
        repository, and you can get details about each of them in
        <xref linkend="svn.ref.reposhooks" />.  As a repository
        administrator, you'll need to decide which hooks you wish
        to implement (by way of providing an appropriately named and
        permissioned hook program), and how.  When you make this
        decision, keep in mind
        the big picture of how your repository is deployed.
        For example, if you are using server configuration
        to determine which users are permitted to commit
        changes to your repository, you don't need to do this
        sort of access control via the hook system.</para>

      <para>There is no shortage of Subversion hook programs and
        scripts that are freely available either from the Subversion community
        itself or elsewhere.  These scripts cover a wide range of
        utility&mdash;basic access control, policy adherence checking,
        issue tracker integration, email- or syndication-based commit
        notification, and beyond.  Or, if you wish to write your own,
        see <xref linkend="svn.developer" />.</para>

      <warning>
        <para>While hook scripts can do almost
          anything, there is one dimension in which hook script
          authors should show restraint:  do <emphasis>not</emphasis>
          modify a commit transaction using hook scripts.  While it
          might be tempting to use hook scripts to automatically
          correct errors, shortcomings, or policy violations present
          in the files being committed, doing so can cause problems.
          Subversion keeps client-side caches of certain bits of
          repository data, and if you change a commit transaction in
          this way, those caches become indetectably stale.  This
          inconsistency can lead to surprising and unexpected
          behavior.  Instead of modifying the transaction, you should
          simply <emphasis>validate</emphasis> the transaction in the
          <filename>pre-commit</filename> hook and reject the commit
          if it does not meet the desired requirements.  As a
          bonus, your users will learn the value of careful,
          compliance-minded work habits.</para>
      </warning>

    </sect2>

    <!-- =============================================================== -->
    <sect2 id="svn.reposadmin.create.bdb">
      <title>Berkeley DB Configuration</title>

      <para>A Berkeley DB environment is an encapsulation of one or
        more databases, logfiles, region files, and configuration
        files.  The Berkeley DB environment has its own set of default
        configuration values for things such as the number of database
        locks allowed to be taken out at any given time, the maximum
        size of the journaling logfiles, and so on.  Subversion's
        filesystem logic additionally chooses default values for some
        of the Berkeley DB configuration options.  However, sometimes
        your particular repository, with its unique collection of data
        and access patterns, might require a different set of
        configuration option values.</para>

      <para>The producers of Berkeley DB understand that different
        applications and database environments have different
        requirements, so they have provided a mechanism for overriding
        at runtime many of the configuration values for the Berkeley
        DB environment.  BDB checks for the presence of a file named
        <filename>DB_CONFIG</filename> in the environment directory
        (namely, the repository's <filename>db</filename>
        subdirectory), and parses the options found in that file.
        Subversion itself creates this file when it creates the rest
        of the repository.  The file initially contains some default
        options, as well as pointers to the Berkeley DB online
        documentation so that you can read about what those options do.  Of
        course, you are free to add any of the supported Berkeley DB
        options to your <filename>DB_CONFIG</filename> file.  Just be
        aware that while Subversion never attempts to read or
        interpret the contents of the file and makes no direct use of
        the option settings in it, you'll want to avoid any
        configuration changes that may cause Berkeley DB to behave in
        a fashion that is at odds with what Subversion might expect.
        Also, changes made to <filename>DB_CONFIG</filename> won't
        take effect until you recover the database environment (using
        <command>svnadmin recover</command>).</para>

    </sect2>
  </sect1>


  <!-- ================================================================= -->
  <!-- ================================================================= -->
  <!-- ================================================================= -->
  <sect1 id="svn.reposadmin.maint">
    <title>Repository Maintenance</title>

    <para>Maintaining a Subversion repository can be daunting, mostly
      due to the complexities inherent in systems that have a database
      backend.  Doing the task well is all about knowing the
      tools&mdash;what they are, when to use them, and how.  This
      section will introduce you to the repository administration
      tools provided by Subversion and discuss how to wield them to
      accomplish tasks such as repository data migration, upgrades,
      backups, and cleanups.</para>

    <!-- =============================================================== -->
    <sect2 id="svn.reposadmin.maint.tk">
      <title>An Administrator's Toolkit</title>

      <para>Subversion provides a handful of utilities useful for
        creating, inspecting, modifying, and repairing your repository.
        Let's look more closely at each of those tools.  Afterward,
        we'll briefly examine some of the utilities included in the
        Berkeley DB distribution that provide functionality specific
        to your repository's database backend not otherwise provided
        by Subversion's own tools.</para>

      <!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -->
      <sect3 id="svn.reposadmin.maint.tk.svnadmin">
        <title>svnadmin</title>

        <para>The <command>svnadmin</command> program is the
          repository administrator's best friend.  Besides providing
          the ability to create Subversion repositories, this program
          allows you to perform several maintenance operations on
          those repositories.  The syntax of
          <command>svnadmin</command> is similar to that of other
          Subversion command-line programs:</para>

        <screen>
$ svnadmin help
general usage: svnadmin SUBCOMMAND REPOS_PATH  [ARGS &amp; OPTIONS ...]
Type 'svnadmin help &lt;subcommand&gt;' for help on a specific subcommand.
Type 'svnadmin --version' to see the program version and FS modules.

Available subcommands:
   crashtest
   create
   deltify
&hellip;
</screen>

        <para>Previously in this chapter (in <xref
          linkend="svn.reposadmin.basics.creating"/>), we were
          introduced to the <command>svnadmin create</command>
          subcommand.  Most of the other <command>svnadmin</command>
          subcommands we will cover later in this chapter.  And you
          can consult <xref linkend="svn.ref.svnadmin" /> for a full
          rundown of subcommands and what each of them offers.</para>

      </sect3>

      <!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -->
      <sect3 id="svn.reposadmin.maint.tk.svnlook">
        <title>svnlook</title>
            
        <para><command>svnlook</command> is a tool provided by
          Subversion for examining the various revisions and
          <firstterm>transactions</firstterm> (which are revisions
          in the making) in a repository.  No part of this program
          attempts to change the repository.  <command>svnlook</command>
          is typically used by the repository hooks for reporting the
          changes that are about to be committed (in the case of the
          <command>pre-commit</command> hook) or that were just
          committed (in the case of the <command>post-commit</command>
          hook) to the repository.  A repository administrator may use
          this tool for diagnostic purposes.</para>
            
        <para><command>svnlook</command> has a straightforward
          syntax:</para>
            
        <screen>
$ svnlook help
general usage: svnlook SUBCOMMAND REPOS_PATH [ARGS &amp; OPTIONS ...]
Note: any subcommand which takes the '--revision' and '--transaction'
      options will, if invoked without one of those options, act on
      the repository's youngest revision.
Type 'svnlook help &lt;subcommand&gt;' for help on a specific subcommand.
Type 'svnlook --version' to see the program version and FS modules.
&hellip;
</screen>


        <para>Most of <command>svnlook</command>'s
          subcommands can operate on either a revision or a
          transaction tree, printing information about the tree
          itself, or how it differs from the previous revision of the
          repository.  You use the <option>--revision</option>
          (<option>-r</option>) and <option>--transaction</option>
          (<option>-t</option>) options to specify which revision or
          transaction, respectively, to examine.  In the absence of
          both the <option>--revision</option> (<option>-r</option>)
          and <option>--transaction</option> (<option>-t</option>)
          options, <command>svnlook</command> will examine the
          youngest (or <literal>HEAD</literal>) revision in the
          repository.  So the following two commands do exactly the
          same thing when 19 is the youngest revision in the
          repository located at
          <filename>/var/svn/repos</filename>:</para>

        <screen>
$ svnlook info /var/svn/repos
$ svnlook info /var/svn/repos -r 19
</screen>

        <para>One exception to these rules about subcommands is
          the <command>svnlook youngest</command> subcommand, which
          takes no options and simply prints out the repository's
          youngest revision number:</para>

        <screen>
$ svnlook youngest /var/svn/repos
19
$
</screen>

        <note>
          <para>Keep in mind that the only transactions you can browse
            are uncommitted ones.  Most repositories will have no such
            transactions because transactions are usually either
            committed (in which case, you should access them as
            revision with the <option>--revision</option>
            (<option>-r</option>) option) or aborted and
            removed.</para>
        </note>
            
        <para>Output from <command>svnlook</command> is designed to be
          both human- and machine-parsable.  Take, as an example, the
          output of the <command>svnlook info</command> subcommand:</para>

        <screen>
$ svnlook info /var/svn/repos
sally
2002-11-04 09:29:13 -0600 (Mon, 04 Nov 2002)
27
Added the usual
Greek tree.
$
</screen>

        <para>The output of <command>svnlook info</command> consists
          of the following, in the order given:</para>

        <orderedlist>
          <listitem>
            <para>The author, followed by a newline</para>
          </listitem>
          <listitem>
            <para>The date, followed by a newline</para>
          </listitem>
          <listitem>
            <para>The number of characters in the log message,
              followed by a newline</para>
          </listitem>
          <listitem>
            <para>The log message itself, followed by a newline</para>
          </listitem>
        </orderedlist>

        <para>This output is human-readable, meaning items such as the
          datestamp are displayed using a textual representation
          instead of something more obscure (such as the number of
          nanoseconds since the Tastee Freez guy drove by).  But the
          output is also machine-parsable&mdash;because the log
          message can contain multiple lines and be unbounded in
          length, <command>svnlook</command> provides the length of
          that message before the message itself.  This allows scripts
          and other wrappers around this command to make intelligent
          decisions about the log message, such as how much memory to
          allocate for the message, or at least how many bytes to skip
          in the event that this output is not the last bit of data in
          the stream.</para>

        <para><command>svnlook</command> can perform a variety of
          other queries:  displaying subsets of bits of information
          we've mentioned previously, recursively listing versioned
          directory trees, reporting which paths were modified in a
          given revision or transaction, showing textual and property
          differences made to files and directories, and so on.  See
          <xref linkend="svn.ref.svnlook" /> for a full reference of
          <command>svnlook</command>'s features.</para>

      </sect3>

      <!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -->
      <sect3 id="svn.reposadmin.maint.tk.svndumpfilter">
        <title>svndumpfilter</title>

        <para>While it won't be the most commonly used tool at the
          administrator's disposal, <command>svndumpfilter</command>
          provides a very particular brand of useful
          functionality&mdash;the ability to quickly and easily modify
          streams of Subversion repository history data by acting as a
          path-based filter.</para>

        <para>The syntax of <command>svndumpfilter</command> is as
          follows:</para>

        <screen>
$ svndumpfilter help
general usage: svndumpfilter SUBCOMMAND [ARGS &amp; OPTIONS ...]
Type "svndumpfilter help &lt;subcommand&gt;" for help on a specific subcommand.
Type 'svndumpfilter --version' to see the program version.
  
Available subcommands:
   exclude
   include
   help (?, h)
</screen>

        <para>There are only two interesting subcommands:
          <command>svndumpfilter exclude</command> and
          <command>svndumpfilter include</command>.  They allow you to
          make the choice between implicit or explicit inclusion of
          paths in the stream.  You can learn more about these
          subcommands and <command>svndumpfilter</command>'s unique
          purpose later in this chapter, in <xref
          linkend="svn.reposadmin.maint.filtering" />.</para>

      </sect3>

      <!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -->
      <sect3 id="svn.reposadmin.maint.tk.svnsync">
        <title>svnsync</title>

        <para>The <command>svnsync</command> program, which is new to
          the 1.4 release of Subversion, provides all the
          functionality required for maintaining a read-only mirror of
          a Subversion repository.  The program really has one
          job&mdash;to transfer one repository's versioned history
          into another repository.  And while there are few ways to do
          that, its primary strength is that it can operate
          remotely&mdash;the <quote>source</quote> and
          <quote>sink</quote>
          <footnote>
            <para>Or is that, the <quote>sync</quote>?</para>
          </footnote>
          repositories may be on different computers from each other
          and from <command>svnsync</command> itself.</para>

        <para>As you might expect, <command>svnsync</command> has a
          syntax that looks very much like every other program we've
          mentioned in this chapter:</para>

        <screen>
$ svnsync help
general usage: svnsync SUBCOMMAND DEST_URL  [ARGS &amp; OPTIONS ...]
Type 'svnsync help &lt;subcommand&gt;' for help on a specific subcommand.
Type 'svnsync --version' to see the program version and RA modules.

Available subcommands:
   initialize (init)
   synchronize (sync)
   copy-revprops
   help (?, h)
$
</screen>

        <para>We talk more about replicating repositories with
          <command>svnsync</command> later in this chapter (see <xref
          linkend="svn.reposadmin.maint.replication" />).</para>

      </sect3>

      <!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -->
      <sect3 id="svn.reposadmin.maint.tk.fsfsreshard">
        <title>fsfs-reshard.py</title>

        <para>While not an official member of the Subversion
          toolchain, the <command>fsfs-reshard.py</command> script
          (found in the <filename>tools/server-side</filename>
          directory of the Subversion source distribution) is a useful
          performance tuning tool for administrators of FSFS-backed
          Subversion repositories.  FSFS repositories contain files
          that describe the changes made in a single revision, and
          files that contain the revision properties associated with
          a single revision.  Repositories created in versions of
          Subversion prior to 1.5 keep these files in two
          directories&mdash;one for each type of file.  As new
          revisions are committed to the repository, Subversion drops
          more files into these two directories&mdash;over time, the
          number of these files in each directory can grow to be quite
          large.  This has been observed to cause performance problems
          on certain network-based filesystems.</para>


        <para>Subversion 1.5 creates FSFS-backed repositories using a
          slightly modified layout in which the contents of these two
          directories are <firstterm>sharded</firstterm>, or scattered
          across several subdirectories.  This can greatly reduce the
          time it takes the system to locate any one of these files,
          and therefore increases the overall performance of
          Subversion when reading from the repository.  The number of
          subdirectories used to house these files is configurable,
          though, and that's where
          <command>fsfs-reshard.py</command> comes in.  This script
          reshuffles the repository's file structure into a new
          arrangement that reflects the requested number of sharding
          subdirectories.  This is especially useful for converting an
          older Subversion repository into the new Subversion 1.5
          sharded layout (which Subversion will not automatically do
          for you) or for fine-tuning an already sharded
          repository.</para>

      </sect3>

      <!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -->
      <sect3 id="svn.reposadmin.maint.tk.bdbutil">
        <title>Berkeley DB utilities</title>

        <para>If you're using a Berkeley DB repository, all of
          your versioned filesystem's structure and data live in a set
          of database tables within the <filename>db/</filename>
          subdirectory of your repository.  This subdirectory is a
          regular Berkeley DB environment directory and can therefore
          be used in conjunction with any of the Berkeley database
          tools, typically provided as part of the Berkeley DB
          distribution.</para>

        <para>For day-to-day Subversion use, these tools are
          unnecessary.  Most of the functionality typically needed for
          Subversion repositories has been duplicated in the
          <command>svnadmin</command> tool.  For example,
          <command>svnadmin list-unused-dblogs</command> and
          <command>svnadmin list-dblogs</command> perform a
          subset of what is provided by the Berkeley
          <command>db_archive</command> utility, and <command>svnadmin
          recover</command> reflects the common use cases of the
          <command>db_recover</command> utility.</para>
            
        <para>However, there are still a few Berkeley DB utilities
          that you might find useful.  The <command>db_dump</command>
          and <command>db_load</command> programs write and read,
          respectively, a custom file format that describes the keys
          and values in a Berkeley DB database.  Since Berkeley
          databases are not portable across machine architectures,
          this format is a useful way to transfer those databases from
          machine to machine, irrespective of architecture or
          operating system.  As we describe later in this chapter, you
          can also use <command>svnadmin dump</command> and
          <command>svnadmin load</command> for similar purposes, but
          <command>db_dump</command> and <command>db_load</command>
          can do certain jobs just as well and much faster.  They can
          also be useful if the experienced Berkeley DB hacker needs
          to do in-place tweaking of the data in a BDB-backed
          repository for some reason, which is something Subversion's
          utilities won't allow.  Also, the <command>db_stat</command>
          utility can provide useful information about the status of
          your Berkeley DB environment, including detailed statistics
          about the locking and storage subsystems.</para>

        <para>For more information on the Berkeley DB tool chain,
          visit the documentation section of the Berkeley DB section
          of Oracle's web site, located at <ulink
          url="http://www.oracle.com/technology/documentation/berkeley-db/db/"
          />.</para>

      </sect3>
    </sect2>

    <!-- =============================================================== -->
    <sect2 id="svn.reposadmin.maint.setlog">
      <title>Commit Log Message Correction</title>
            
      <para>Sometimes a user will have an error in her log message (a
        misspelling or some misinformation, perhaps).  If the
        repository is configured (using the
        <literal>pre-revprop-change</literal> hook; see <xref
        linkend="svn.reposadmin.create.hooks"/>) to accept changes to
        this log message after the commit is finished, the user
        can <quote>fix</quote> her log message remotely using
        <command>svn propset</command> (see <xref
        linkend="svn.ref.svn.c.propset"/>).  However, because of the
        potential to lose information forever, Subversion repositories
        are not, by default, configured to allow changes to
        unversioned properties&mdash;except by an
        administrator.</para>

      <para>If a log message needs to be changed by an administrator,
        this can be done using <command>svnadmin setlog</command>.
        This command changes the log message (the
        <literal>svn:log</literal> property) on a given revision of a
        repository, reading the new value from a provided file.</para>
          
      <screen>
$ echo "Here is the new, correct log message" &gt; newlog.txt
$ svnadmin setlog myrepos newlog.txt -r 388
</screen>
      
      <para>The <command>svnadmin setlog</command> command, by
        default, is
        still bound by the same protections against modifying
        unversioned properties as a remote client is&mdash;the
        <literal>pre-</literal> and
        <literal>post-revprop-change</literal> hooks are still
        triggered, and therefore must be set up to accept changes of
        this nature.  But an administrator can get around these
        protections by passing the <option>--bypass-hooks</option>
        option to the <command>svnadmin setlog</command> command.</para>
 
      <warning>
        <para>Remember, though, that by bypassing the hooks, you are
          likely avoiding such things as email notifications of
          property changes, backup systems that track unversioned
          property changes, and so on.  In other words, be very
          careful about what you are changing, and how you change
          it.</para>
      </warning>


    </sect2>

    <!-- =============================================================== -->
    <sect2 id="svn.reposadmin.maint.diskspace">
      <title>Managing Disk Space</title>

      <para>While the cost of storage has dropped incredibly in the
        past few years, disk usage is still a valid concern for
        administrators seeking to version large amounts of data.
        Every bit of version history information stored in the live
        repository needs to be backed up
        elsewhere, perhaps multiple times as part of rotating backup
        schedules.  It is useful to know what pieces of Subversion's
        repository data need to remain on the live site, which need to
        be backed up, and which can be safely removed.</para>

      <!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -->
      <sect3 id="svn.reposadmin.maint.diskspace.deltas">
        <title>How Subversion saves disk space</title>

        <para>To keep the repository small,
          Subversion uses <firstterm>deltification</firstterm> (or
          deltified storage) within the repository
          itself.  Deltification involves encoding the representation
          of a chunk of data as a collection of differences against
          some other chunk of data.  If the two pieces of data are
          very similar, this deltification results in storage savings
          for the deltified chunk&mdash;rather than taking up space
          equal to the size of the original data, it takes up only
          enough space to say, <quote>I look just like this other
          piece of data over here, except for the following couple of
          changes.</quote>  The result is that most of the repository
          data that tends to be bulky&mdash;namely, the contents of
          versioned files&mdash;is stored at a much smaller size than
          the original full-text representation of that
          data.  And for repositories created with Subversion 1.4 or
          later, the space savings are even better&mdash;now those
          full-text representations of file contents are themselves
          compressed.</para>

        <note>
          <para>Because all of the data that is subject to
            deltification in a BDB-backed repository is stored in a
            single Berkeley DB database file, reducing the size of the
            stored values will not immediately reduce the size of the
            database file itself.  Berkeley DB will, however, keep
            internal records of unused areas of the database file and
            consume those areas first before growing the size of the
            database file.  So while deltification doesn't produce
            immediate space savings, it can drastically slow future
            growth of the database.</para>
        </note>

      </sect3>

      <!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -->
      <sect3 id="svn.reposadmin.maint.diskspace.deadtxns">
        <title>Removing dead transactions</title>

        <para>Though they are uncommon, there are circumstances in
          which a Subversion commit process might fail, leaving behind
          in the repository the remnants of the revision-to-be that
          wasn't&mdash;an uncommitted transaction and all the file and
          directory changes associated with it.  This could happen for
          several reasons:  perhaps the client operation was
          inelegantly terminated by the user, or a network failure
          occurred in the middle of an operation.
          Regardless of the reason, dead transactions can happen.
          They don't do any real harm, other than consuming disk
          space.  A fastidious administrator may nonetheless wish to
          remove them.</para>

        <para>You can use the <command>svnadmin lstxns</command>
          command to list the names of the currently outstanding
          transactions:</para>


        <screen>
$ svnadmin lstxns myrepos
19
3a1
a45
$
</screen>

        <para>Each item in the resultant output can then be used with
          <command>svnlook</command> (and its
          <option>--transaction</option> (<option>-t</option>) option)
          to determine who created the transaction, when it was
          created, what types of changes were made in the
          transaction&mdash;information that is helpful in determining
          whether the transaction is a safe candidate for
          removal!  If you do indeed want to remove a transaction, its
          name can be passed to <command>svnadmin rmtxns</command>,
          which will perform the cleanup of the transaction.  In fact,
          <command>svnadmin rmtxns</command> can take its input
          directly from the output of
          <command>svnadmin lstxns</command>!</para>

        <screen>
$ svnadmin rmtxns myrepos `svnadmin lstxns myrepos`
$
</screen>

        <para>If you use these two subcommands like this, you should
          consider making your repository temporarily inaccessible to
          clients.  That way, no one can begin a legitimate
          transaction before you start your cleanup.  <xref
          linkend="svn.reposadmin.maint.diskspace.deadtxns.ex-1" />
          contains a bit of shell-scripting that can quickly generate
          information about each outstanding transaction in your
          repository.</para>

        <example id="svn.reposadmin.maint.diskspace.deadtxns.ex-1">
          <title>txn-info.sh (reporting outstanding transactions)</title>

          <programlisting>
#!/bin/sh

### Generate informational output for all outstanding transactions in
### a Subversion repository.

REPOS="${1}"
if [ "x$REPOS" = x ] ; then
  echo "usage: $0 REPOS_PATH"
  exit
fi

for TXN in `svnadmin lstxns ${REPOS}`; do 
  echo "---[ Transaction ${TXN} ]-------------------------------------------"
  svnlook info "${REPOS}" -t "${TXN}"
done
</programlisting>
        </example>

        <para>The output of the script is basically a concatenation of
          several chunks of <command>svnlook info</command> output
          (see <xref linkend="svn.reposadmin.maint.tk.svnlook"/>) and
          will look something like this:</para>

        <screen>
$ txn-info.sh myrepos
---[ Transaction 19 ]-------------------------------------------
sally
2001-09-04 11:57:19 -0500 (Tue, 04 Sep 2001)
0
---[ Transaction 3a1 ]-------------------------------------------
harry
2001-09-10 16:50:30 -0500 (Mon, 10 Sep 2001)
39
Trying to commit over a faulty network.
---[ Transaction a45 ]-------------------------------------------
sally
2001-09-12 11:09:28 -0500 (Wed, 12 Sep 2001)
0
$
</screen>

        <para>A long-abandoned transaction usually represents some
          sort of failed or interrupted commit.  A transaction's
          datestamp can provide interesting information&mdash;for
          example, how likely is it that an operation begun nine
          months ago is still active?</para>

        <para>In short, transaction cleanup decisions need not be made
          unwisely.  Various sources of information&mdash;including
          Apache's error and access logs, Subversion's operational
          logs, Subversion revision history, and so on&mdash;can be
          employed in the decision-making process.  And of course, an
          administrator can often simply communicate with a seemingly
          dead transaction's owner (via email, e.g.) to verify
          that the transaction is, in fact, in a zombie state.</para>

      </sect3>

      <!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -->
      <sect3 id="svn.reposadmin.maint.diskspace.bdblogs">
        <title>Purging unused Berkeley DB logfiles</title>

        <para>Until recently, the largest offender of disk space usage
          with respect to BDB-backed Subversion repositories were the
          logfiles in which Berkeley DB performs its prewrites before
          modifying the actual database files.  These files capture
          all the actions taken along the route of changing the
          database from one state to another&mdash;while the database
          files, at any given time, reflect a particular state, the
          logfiles contain all of the many changes along the way
          <emphasis>between</emphasis> states.  Thus, they can grow
          and accumulate quite rapidly.</para>

        <para>Fortunately, beginning with the 4.2 release of Berkeley
          DB, the database environment has the ability to remove its
          own unused logfiles automatically.  Any
          repositories created using <command>svnadmin</command>
          when compiled against Berkeley DB version 4.2 or later
          will be configured for this automatic logfile removal.  If
          you don't want this feature enabled, simply pass the
          <option>--bdb-log-keep</option> option to the
          <command>svnadmin create</command> command.  If you forget
          to do this or change your mind at a later time, simply edit
          the <filename>DB_CONFIG</filename> file found in your
          repository's <filename>db</filename> directory, comment out
          the line that contains the <literal>set_flags
          DB_LOG_AUTOREMOVE</literal> directive, and then run
          <command>svnadmin recover</command> on your repository to
          force the configuration changes to take effect.  See <xref
          linkend="svn.reposadmin.create.bdb"/> for more information about
          database configuration.</para>

        <para>Without some sort of automatic logfile removal in
          place, logfiles will accumulate as you use your repository.
          This is actually somewhat of a feature of the database
          system&mdash;you should be able to recreate your entire
          database using nothing but the logfiles, so these files can
          be useful for catastrophic database recovery.  But
          typically, you'll want to archive the logfiles that are no
          longer in use by Berkeley DB, and then remove them from disk
          to conserve space.  Use the <command>svnadmin
          list-unused-dblogs</command> command to list the unused
          logfiles:</para>

        <screen>
$ svnadmin list-unused-dblogs /var/svn/repos
/var/svn/repos/log.0000000031
/var/svn/repos/log.0000000032
/var/svn/repos/log.0000000033
&hellip;
$ rm `svnadmin list-unused-dblogs /var/svn/repos`
## disk space reclaimed!
</screen>

        <warning>
          <para>BDB-backed repositories whose logfiles are used as
            part of a backup or disaster recovery plan should
            <emphasis>not</emphasis> make use of the logfile
            autoremoval feature.  Reconstruction of a repository's
            data from logfiles can only be accomplished only when
            <emphasis>all</emphasis> the logfiles are available.  If
            some of the logfiles are removed from disk before the
            backup system has a chance to copy them elsewhere, the
            incomplete set of backed-up logfiles is essentially
            useless.</para> </warning>

      </sect3>

    </sect2>
        
    <!-- =============================================================== -->
    <sect2 id="svn.reposadmin.maint.recovery">
      <title>Berkeley DB Recovery</title>

      <para>As mentioned in <xref
        linkend="svn.reposadmin.basics.backends.bdb"/>, a Berkeley DB
        repository can sometimes be left in a frozen state if not closed
        properly.  When this happens, an administrator needs to rewind
        the database back into a consistent state.  This is unique to
        BDB-backed repositories, though&mdash;if you are using
        FSFS-backed ones instead, this won't apply to you.  And for
        those of you using Subversion 1.4 with Berkeley DB 4.4 or
        later, you should find that Subversion has become much more
        resilient in these types of situations.  Still, wedged
        Berkeley DB repositories do occur, and an administrator needs
        to know how to safely deal with this circumstance.</para>

      <para>To protect the data in your repository, Berkeley
        DB uses a locking mechanism.  This mechanism ensures that
        portions of the database are not simultaneously modified by
        multiple database accessors, and that each process sees the
        data in the correct state when that data is being read from
        the database.  When a process needs to change something in the
        database, it first checks for the existence of a lock on the
        target data.  If the data is not locked, the process locks the
        data, makes the change it wants to make, and then unlocks the
        data.  Other processes are forced to wait until that lock is
        removed before they are permitted to continue accessing that
        section of the database.  (This has nothing to do with the
        locks that you, as a user, can apply to versioned files within
        the repository; we try to clear up the confusion caused by
        this terminology collision in the sidebar <xref
        linkend="svn.advanced.locking.meanings" />.)</para>


      <para>In the course of using your Subversion repository, fatal
        errors or interruptions can prevent a process from having the
        chance to remove the locks it has placed in the database.  The
        result is that the backend database system gets
        <quote>wedged.</quote>  When this happens, any attempts to
        access the repository hang indefinitely (since each new
        accessor is waiting for a lock to go away&mdash;which isn't
        going to happen).</para>

      <para>If this happens to your repository, don't panic.  The
        Berkeley DB filesystem takes advantage of database
        transactions, checkpoints, and prewrite journaling to
        ensure that only the most catastrophic of events
        <footnote>
          <para>For example, hard drive + huge electromagnet = disaster.</para>
        </footnote>
        can permanently destroy a database environment.  A
        sufficiently paranoid repository administrator will have made
        off-site backups of the repository data in some fashion, but
        don't head off to the tape backup storage closet just yet.</para>

      <para>Instead, use the following recipe to attempt to
        <quote>unwedge</quote> your repository:</para>
   
      <orderedlist>
        <listitem>
          <para>Make sure no processes are accessing (or
            attempting to access) the repository.  For networked
            repositories, this also means shutting down the Apache HTTP
            Server or svnserve daemon.</para>
        </listitem>
        <listitem> 
          <para>Become the user who owns and manages the repository.
            This is important, as recovering a repository while
            running as the wrong user can tweak the permissions of the
            repository's files in such a way that your repository will
            still be inaccessible even after it is 
            <quote>unwedged.</quote></para>
        </listitem>
        <listitem>
          <para>Run the command <userinput>svnadmin recover
            /var/svn/repos</userinput>.  You should see output such as
            this:</para>
              
          <screen>
Repository lock acquired.
Please wait; recovering the repository may take some time...

Recovery completed.
The latest repos revision is 19.
</screen>
          <para>This command may take many minutes to complete.</para>
        </listitem>
        <listitem>
          <para>Restart the server process.</para>
        </listitem>
      </orderedlist>
            
      <para>This procedure fixes almost every case of repository
        wedging.  Make sure that you run this command as the user that
        owns and manages the database, not just as
        <literal>root</literal>.  Part of the recovery process might
        involve re-creating from scratch various database files (shared
        memory regions, e.g.).  Recovering as
        <literal>root</literal> will create those files such that they
        are owned by <literal>root</literal>, which means that even
        after you restore connectivity to your repository, regular
        users will be unable to access it.</para>

      <para>If the previous procedure, for some reason, does not
        successfully unwedge your repository, you should do two
        things.  First, move your broken repository directory aside
        (perhaps by renaming it to something like
        <filename>repos.BROKEN</filename>) and then restore your
        latest backup of it.  Then, send an email to the Subversion
        users mailing list (at <email>users@subversion.tigris.org</email>)
        describing your problem in detail.  Data integrity is an
        extremely high priority to the Subversion developers.</para>

    </sect2>

    <!-- =============================================================== -->
    <sect2 id="svn.reposadmin.maint.migrate">
      <title>Migrating Repository Data Elsewhere</title>
    
      <para>A Subversion filesystem has its data spread throughout
        files in the repository, in a fashion generally
        understood by (and of interest to) only the Subversion
        developers themselves.  However, circumstances may arise that
        call for all, or some subset, of that data to be copied or
        moved into another repository.</para>

      <para>Subversion provides such functionality by way of
        <firstterm>repository dump streams</firstterm>.  A repository
        dump stream (often referred to as a <quote>dump file</quote>
        when stored as a file on disk) is a portable, flat file format
        that describes the various revisions in your
        repository&mdash;what was changed, by whom, when, and so on.
        This dump stream is the primary mechanism used to marshal
        versioned history&mdash;in whole or in part, with or without
        modification&mdash;between repositories.  And Subversion
        provides the tools necessary for creating and loading these
        dump streams: the <command>svnadmin dump</command> and
        <command>svnadmin load</command> subcommands,
        respectively.</para>

      <warning>
        <para>While the Subversion repository dump format contains
          human-readable portions and a familiar structure (it
          resembles an RFC 822 format, the same type of format used
          for most email), it is <emphasis>not</emphasis> a plain-text
          file format.  It is a binary file format, highly sensitive
          to meddling.  For example, many text editors will corrupt
          the file by automatically converting line endings.</para>
      </warning>

      <para>There are many reasons for dumping and loading Subversion
        repository data.  Early in Subversion's life, the most common
        reason was due to the evolution of Subversion itself.  As
        Subversion matured, there were times when changes made to the
        backend database schema caused compatibility issues with
        previous versions of the repository, so users had to dump
        their repository data using the previous version of
        Subversion and load it into a freshly created repository with
        the new version of Subversion.  Now, these types of schema
        changes haven't occurred since Subversion's 1.0 release, and
        the Subversion developers promise not to force users to dump
        and load their repositories when upgrading between minor
        versions (such as from 1.3 to 1.4) of Subversion.  But there
        are still other reasons for dumping and loading, including
        re-deploying a Berkeley DB repository on a new OS or CPU
        architecture, switching between the Berkeley DB and FSFS
        backends, or (as we'll cover later in this chapter in <xref
        linkend="svn.reposadmin.maint.filtering" />) purging versioned
        data from repository history.</para>

      <note>
        <para>The Subversion repository dump format describes
          versioned repository changes only.  It will not carry any
          information about uncommitted transactions, user locks on
          filesystem paths, repository or server configuration
          customizations (including hook scripts), and so on.</para>
      </note>

      <para>Whatever your reason for migrating repository history,
        using the <command>svnadmin dump</command> and
        <command>svnadmin load</command> subcommands is
        straightforward.  <command>svnadmin dump</command> will output
        a range of repository revisions that are formatted using
        Subversion's custom filesystem dump format.  The dump format
        is printed to the standard output stream, while informative
        messages are printed to the standard error stream.  This
        allows you to redirect the output stream to a file while
        watching the status output in your terminal window.  For
        example:</para>

      <screen>
$ svnlook youngest myrepos
26
$ svnadmin dump myrepos &gt; dumpfile
* Dumped revision 0.
* Dumped revision 1.
* Dumped revision 2.
&hellip;
* Dumped revision 25.
* Dumped revision 26.
</screen>

      <para>At the end of the process, you will have a single file
        (<filename>dumpfile</filename> in the previous example) that
        contains all the data stored in your repository in the
        requested range of revisions.  Note that <command>svnadmin
        dump</command> is reading revision trees from the repository
        just like any other <quote>reader</quote> process would
        (e.g., <command>svn checkout</command>), so it's safe
        to run this command at any time.</para>

      <para>The other subcommand in the pair, <command>svnadmin
        load</command>, parses the standard input stream as a
        Subversion repository dump file and effectively replays those
        dumped revisions into the target repository for that
        operation.  It also gives informative feedback, this time
        using the standard output stream:</para>

      <screen>
$ svnadmin load newrepos &lt; dumpfile
&lt;&lt;&lt; Started new txn, based on original revision 1
     * adding path : A ... done.
     * adding path : A/B ... done.
     &hellip;
------- Committed new rev 1 (loaded from original rev 1) &gt;&gt;&gt;

&lt;&lt;&lt; Started new txn, based on original revision 2
     * editing path : A/mu ... done.
     * editing path : A/D/G/rho ... done.

------- Committed new rev 2 (loaded from original rev 2) &gt;&gt;&gt;

&hellip;


&lt;&lt;&lt; Started new txn, based on original revision 25
     * editing path : A/D/gamma ... done.

------- Committed new rev 25 (loaded from original rev 25) &gt;&gt;&gt;

&lt;&lt;&lt; Started new txn, based on original revision 26
     * adding path : A/Z/zeta ... done.
     * editing path : A/mu ... done.

------- Committed new rev 26 (loaded from original rev 26) &gt;&gt;&gt;

</screen>

      <para>The result of a load is new revisions added to a
        repository&mdash;the same thing you get by making commits
        against that repository from a regular Subversion client.
        Just as in a commit, you can use hook programs to perform
        actions before and after each of the commits made during a
        load process.  By passing the
        <option>--use-pre-commit-hook</option> and
        <option>--use-post-commit-hook</option> options to
        <command>svnadmin load</command>, you can instruct Subversion
        to execute the pre-commit and post-commit hook programs,
        respectively, for each loaded revision.  You might use these,
        for example, to ensure that loaded revisions pass through the
        same validation steps that regular commits pass through.  Of
        course, you should use these options with care&mdash;if your
        post-commit hook sends emails to a mailing list for each new
        commit, you might not want to spew hundreds or thousands of
        commit emails in rapid succession at that list!  You can read more about the use of hook
        scripts in <xref
        linkend="svn.reposadmin.create.hooks"/>.</para>

      <para>Note that because <command>svnadmin</command> uses
        standard input and output streams for the repository dump and
        load processes, people who are feeling especially saucy can try
        things such as this (perhaps even using different versions of
        <command>svnadmin</command> on each side of the pipe):</para>
  
      <screen>
$ svnadmin create newrepos
$ svnadmin dump oldrepos | svnadmin load newrepos
</screen>

      <para>By default, the dump file will be quite large&mdash;much
        larger than the repository itself.  That's because by default
        every version of every file is expressed as a full text in the
        dump file.  This is the fastest and simplest behavior, and
        it's nice if you're piping the dump data directly into some other
        process (such as a compression program, filtering program, or
        loading process).  But if you're creating a dump file
        for longer-term storage, you'll likely want to save disk space
        by using the <option>--deltas</option> option.  With this
        option, successive revisions of files will be output as
        compressed, binary differences&mdash;just as file revisions
        are stored in a repository.  This option is slower, but it
        results in a dump file much closer in size to the original
        repository.</para>

      <para>We mentioned previously that <command>svnadmin
        dump</command> outputs a range of revisions.  Use the
        <option>--revision</option> (<option>-r</option>) option to
        specify a single revision, or a range of revisions, to dump.
        If you omit this option, all the existing repository revisions
        will be dumped.</para>

      <screen>
$ svnadmin dump myrepos -r 23 &gt; rev-23.dumpfile
$ svnadmin dump myrepos -r 100:200 &gt; revs-100-200.dumpfile
</screen>

      <para>As Subversion dumps each new revision, it outputs only
        enough information to allow a future loader to re-create that
        revision based on the previous one.  In other words, for any
        given revision in the dump file, only the items that were
        changed in that revision will appear in the dump.  The only
        exception to this rule is the first revision that is dumped
        with the current <command>svnadmin dump</command>
        command.</para>

      <para>By default, Subversion will not express the first dumped
        revision as merely differences to be applied to the previous
        revision.  For one thing, there is no previous revision in the
        dump file!  And second, Subversion cannot know the state of
        the repository into which the dump data will be loaded (if it
        ever is).  To ensure that the output of each
        execution of <command>svnadmin dump</command> is
        self-sufficient, the first dumped revision is, by default, a
        full representation of every directory, file, and property in
        that revision of the repository.</para>

      <para>However, you can change this default behavior.  If you add
        the <option>--incremental</option> option when you dump your
        repository, <command>svnadmin</command> will compare the first
        dumped revision against the previous revision in the
        repository&mdash;the same way it treats every other revision that
        gets dumped.  It will then output the first revision exactly
        as it does the rest of the revisions in the dump
        range&mdash;mentioning only the changes that occurred in that
        revision.  The benefit of this is that you can create several
        small dump files that can be loaded in succession, instead of
        one large one, like so:</para>

      <screen>
$ svnadmin dump myrepos -r 0:1000 &gt; dumpfile1
$ svnadmin dump myrepos -r 1001:2000 --incremental &gt; dumpfile2
$ svnadmin dump myrepos -r 2001:3000 --incremental &gt; dumpfile3
</screen>

      <para>These dump files could be loaded into a new repository
        with the following command sequence:</para>

      <screen>
$ svnadmin load newrepos &lt; dumpfile1
$ svnadmin load newrepos &lt; dumpfile2
$ svnadmin load newrepos &lt; dumpfile3
</screen>

      <para>Another neat trick you can perform with this
        <option>--incremental</option> option involves appending to an
        existing dump file a new range of dumped revisions.  For
        example, you might have a <literal>post-commit</literal> hook
        that simply appends the repository dump of the single revision
        that triggered the hook.  Or you might have a script that runs
        nightly to append dump file data for all the revisions that
        were added to the repository since the last time the script
        ran.  Used like this, <command>svnadmin dump</command> can be
        one way to back up changes to your repository over time in case
        of a system crash or some other catastrophic event.</para>

      <para>The dump format can also be used to merge the contents of
        several different repositories into a single repository.  By
        using the <option>--parent-dir</option> option of
        <command>svnadmin load</command>, you can specify a new
        virtual root directory for the load process.  That means if
        you have dump files for three repositories&mdash;say
        <filename>calc-dumpfile</filename>,
        <filename>cal-dumpfile</filename>, and
        <filename>ss-dumpfile</filename>&mdash;you can first create a new
        repository to hold them all:</para>

      <screen>
$ svnadmin create /var/svn/projects
$
</screen>

      <para>Then, make new directories in the repository that will
        encapsulate the contents of each of the three previous
        repositories:</para>

      <screen>
$ svn mkdir -m "Initial project roots" \
      file:///var/svn/projects/calc \
      file:///var/svn/projects/calendar \
      file:///var/svn/projects/spreadsheet
Committed revision 1.
$ 
</screen>

      <para>Lastly, load the individual dump files into their
        respective locations in the new repository:</para>

      <screen>
$ svnadmin load /var/svn/projects --parent-dir calc &lt; calc-dumpfile
&hellip;
$ svnadmin load /var/svn/projects --parent-dir calendar &lt; cal-dumpfile
&hellip;
$ svnadmin load /var/svn/projects --parent-dir spreadsheet &lt; ss-dumpfile
&hellip;
$
</screen>

      <para>We'll mention one final way to use the Subversion
        repository dump format&mdash;conversion from a different
        storage mechanism or version control system altogether.
        Because the dump file format is, for the most part,
        human-readable, it should be relatively easy to describe
        generic sets of changes&mdash;each of which should be treated
        as a new revision&mdash;using this file format.  In fact, the
        <command>cvs2svn</command> utility (see <xref
        linkend="svn.forcvs.convert"/>) uses the dump format to
        represent the contents of a CVS repository so that those
        contents can be copied into a Subversion repository.</para>

    </sect2>

    <!-- =============================================================== -->
    <sect2 id="svn.reposadmin.maint.filtering">
      <title>Filtering Repository History</title>

      <para>Since Subversion stores your versioned history using, at
        the very least, binary differencing algorithms and data
        compression (optionally in a completely opaque database
        system), attempting manual tweaks is unwise if not quite
        difficult, and at any rate strongly discouraged.  And once
        data has been stored in your repository, Subversion
        generally doesn't provide an easy way to remove that data.
        <footnote>
          <para>That's rather the reason you use version control at
            all, right?</para>
        </footnote>
        But inevitably, there will be times when you would like to
        manipulate the history of your repository.  You might need
        to strip out all instances of a file that was accidentally
        added to the repository (and shouldn't be there for whatever
        reason).
        <footnote>
          <para>Conscious, cautious removal of certain bits of
            versioned data is actually supported by real use cases.
            That's why an <quote>obliterate</quote> feature has been
            one of the most highly requested Subversion features,
            and one which the Subversion developers hope to soon
            provide.</para>
        </footnote>
        Or, perhaps you have multiple projects sharing a
        single repository, and you decide to split them up into
        their own repositories.  To accomplish tasks such as these,
        administrators need a more manageable and malleable
        representation of the data in their repositories&mdash;the
        Subversion repository dump format.</para>


      <para>As we described earlier in <xref
        linkend="svn.reposadmin.maint.migrate" />, the Subversion
        repository dump format is a human-readable representation of
        the changes that you've made to your versioned data over time.
        Use the <command>svnadmin dump</command> command to generate
        the dump data, and <command>svnadmin load</command> to
        populate a new repository with it.  The great thing about the
        human-readability aspect of the dump format is that, if you
        aren't careless about it, you can manually inspect and modify
        it.  Of course, the downside is that if you have three years'
        worth of repository activity encapsulated in what is likely to
        be a very large dump file, it could take you a long, long time
        to manually inspect and modify it.</para>

      <para>That's where <command>svndumpfilter</command> becomes
        useful.  This program acts as a path-based filter for
        repository dump streams.  Simply give it either a list of
        paths you wish to keep or a list of paths you wish to not
        keep, and then pipe your repository dump data through this
        filter.  The result will be a modified stream of dump data
        that contains only the versioned paths you (explicitly or
        implicitly) requested.</para>

      <para>Let's look at a realistic example of how you might use this
        program.  Earlier in this chapter (see <xref
        linkend="svn.reposadmin.projects.chooselayout"/>), we discussed the
        process of deciding how to choose a layout for the data in
        your repositories&mdash;using one repository per project or
        combining them, arranging stuff within your repository, and
        so on.  But sometimes after new revisions start flying in,
        you rethink your layout and would like to make some changes.
        A common change is the decision to move multiple projects
        that are sharing a single repository into separate
        repositories for each project.</para>

      <para>Our imaginary repository contains three projects:
        <literal>calc</literal>, <literal>calendar</literal>, and
        <literal>spreadsheet</literal>.  They have been living
        side-by-side in a layout like this:</para>

      <screen>
/
   calc/
      trunk/
      branches/
      tags/
   calendar/
      trunk/
      branches/
      tags/
   spreadsheet/
      trunk/
      branches/
      tags/
</screen>

      <para>To get these three projects into their own repositories,
        we first dump the whole repository:</para>

      <screen>
$ svnadmin dump /var/svn/repos &gt; repos-dumpfile
* Dumped revision 0.
* Dumped revision 1.
* Dumped revision 2.
* Dumped revision 3.
&hellip;
$
</screen>

      <para>Next, run that dump file through the filter, each time
        including only one of our top-level directories.  This results
        in three new dump files:</para>

      <screen>
$ svndumpfilter include calc &lt; repos-dumpfile &gt; calc-dumpfile
&hellip;
$ svndumpfilter include calendar &lt; repos-dumpfile &gt; cal-dumpfile
&hellip;
$ svndumpfilter include spreadsheet &lt; repos-dumpfile &gt; ss-dumpfile
&hellip;
$
</screen>

      <para>At this point, you have to make a decision.  Each of your
        dump files will create a valid repository, but will preserve
        the paths exactly as they were in the original repository.
        This means that even though you would have a repository solely
        for your <literal>calc</literal> project, that repository
        would still have a top-level directory named
        <filename>calc</filename>.  If you want your
        <filename>trunk</filename>, <filename>tags</filename>, and
        <filename>branches</filename> directories to live in the root
        of your repository, you might wish to edit your dump files,
        tweaking the <literal>Node-path</literal> and
        <literal>Node-copyfrom-path</literal> headers so that they no
        longer have that first <filename>calc/</filename> path
        component.  Also, you'll want to remove the section of dump
        data that creates the <filename>calc</filename> directory.  It
        will look something like the following:</para>

      <screen>
Node-path: calc
Node-action: add
Node-kind: dir
Content-length: 0
  
</screen>

      <warning>
        <para>If you do plan on manually editing the dump file to
          remove a top-level directory, make sure your editor is
          not set to automatically convert end-of-line characters to
          the native format (e.g., <literal>\r\n</literal> to
          <literal>\n</literal>), as the content will then not agree
          with the metadata.  This will render the dump file
          useless.</para>
      </warning>

      <para>All that remains now is to create your three new
        repositories, and load each dump file into the right
        repository, ignoring the UUID found in the dump stream:</para>

      <screen>
$ svnadmin create calc
$ svnadmin load --ignore-uuid calc &lt; calc-dumpfile
&lt;&lt;&lt; Started new transaction, based on original revision 1
     * adding path : Makefile ... done.
     * adding path : button.c ... done.
&hellip;
$ svnadmin create calendar
$ svnadmin load --ignore-uuid calendar &lt; cal-dumpfile
&lt;&lt;&lt; Started new transaction, based on original revision 1
     * adding path : Makefile ... done.
     * adding path : cal.c ... done.
&hellip;
$ svnadmin create spreadsheet
$ svnadmin load --ignore-uuid spreadsheet &lt; ss-dumpfile
&lt;&lt;&lt; Started new transaction, based on original revision 1
     * adding path : Makefile ... done.
     * adding path : ss.c ... done.
&hellip;
$
</screen>

      <para>Both of <command>svndumpfilter</command>'s subcommands
        accept options for deciding how to deal with
        <quote>empty</quote> revisions.  If a given revision
        contains only changes to paths that were filtered out, that
        now-empty revision could be considered uninteresting or even
        unwanted.  So to give the user control over what to do with
        those revisions, <command>svndumpfilter</command> provides
        the following command-line options:</para>

      <variablelist>
        <varlistentry>
          <term><option>--drop-empty-revs</option></term>
          <listitem>
            <para>Do not generate empty revisions at all&mdash;just
              omit them.</para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>--renumber-revs</option></term>
          <listitem>
            <para>If empty revisions are dropped (using the
              <option>--drop-empty-revs</option> option), change the
              revision numbers of the remaining revisions so that
              there are no gaps in the numeric sequence.</para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>--preserve-revprops</option></term>
          <listitem>
            <para>If empty revisions are not dropped, preserve the
              revision properties (log message, author, date, custom
              properties, etc.) for those empty revisions.
              Otherwise, empty revisions will contain only the
              original datestamp, and a generated log message that
              indicates that this revision was emptied by
              <command>svndumpfilter</command>.</para>
          </listitem>
        </varlistentry>
      </variablelist>
      
      <para>While <command>svndumpfilter</command> can be very
        useful and a huge timesaver, there are unfortunately a
        couple of gotchas.  First, this utility is overly sensitive
        to path semantics.  Pay attention to whether paths in your
        dump file are specified with or without leading slashes.
        You'll want to look at the <literal>Node-path</literal> and
        <literal>Node-copyfrom-path</literal> headers.</para>

      <screen>
&hellip;
Node-path: spreadsheet/Makefile
&hellip;
</screen>

      <para>If the paths have leading slashes, you should
        include leading slashes in the paths you pass to
        <command>svndumpfilter include</command> and
        <command>svndumpfilter exclude</command> (and if they don't,
        you shouldn't).  Further, if your dump file has an inconsistent
        usage of leading slashes for some reason,
        <footnote>
          <para>While <command>svnadmin dump</command> has a
            consistent leading slash policy (to not include
            them), other programs that generate dump data might
            not be so consistent.</para>
        </footnote>
        you should probably normalize those paths so that they all
        have, or all lack, leading slashes.</para>


      <para>Also, copied paths can give you some trouble.
        Subversion supports copy operations in the repository, where
        a new path is created by copying some already existing path.
        It is possible that at some point in the lifetime of your
        repository, you might have copied a file or directory from
        some location that <command>svndumpfilter</command> is
        excluding, to a location that it is including.  To
        make the dump data self-sufficient,
        <command>svndumpfilter</command> needs to still show the
        addition of the new path&mdash;including the contents of any
        files created by the copy&mdash;and not represent that
        addition as a copy from a source that won't exist in your
        filtered dump data stream.  But because the Subversion
        repository dump format shows only what was changed in each
        revision, the contents of the copy source might not be
        readily available.  If you suspect that you have any copies
        of this sort in your repository, you might want to rethink
        your set of included/excluded paths, perhaps including the
        paths that served as sources of your troublesome copy
        operations, too.</para>

      <para>Finally, <command>svndumpfilter</command> takes path
        filtering quite literally.  If you are trying to copy the
        history of a project rooted at
        <filename>trunk/my-project</filename> and move it into a
        repository of its own, you would, of course, use the
        <command>svndumpfilter include</command> command to keep all
        the changes in and under
        <filename>trunk/my-project</filename>.  But the resultant
        dump file makes no assumptions about the repository into
        which you plan to load this data.  Specifically, the dump
        data might begin with the revision that added the
        <filename>trunk/my-project</filename> directory, but it will
        <emphasis>not</emphasis> contain directives that would
        create the <filename>trunk</filename> directory itself
        (because <filename>trunk</filename> doesn't match the
        include filter).  You'll need to make sure that any
        directories that the new dump stream expects to exist
        actually do exist in the target repository before trying to
        load the stream into that repository.</para>

    </sect2>
  
    <!-- =============================================================== -->
    <sect2 id="svn.reposadmin.maint.replication">
      <title>Repository Replication</title>

      <para>There are several scenarios in which it is quite handy to
        have a Subversion repository whose version history is exactly
        the same as some other repository's.  Perhaps the most obvious
        one is the maintenance of a simple backup repository, used
        when the primary repository has become inaccessible due to a
        hardware failure, network outage, or other such annoyance.
        Other scenarios include deploying mirror repositories to
        distribute heavy Subversion load across multiple servers, use
        as a soft-upgrade mechanism, and so on.</para>

      <para>As of version 1.4, Subversion provides a program for
        managing scenarios such as
        these&mdash;<command>svnsync</command>.  This works by
        essentially asking the Subversion server to
        <quote>replay</quote> revisions, one at a time.  It then uses
        that revision information to mimic a commit of the same to
        another repository.  Neither repository needs to be locally
        accessible to the machine on which <command>svnsync</command> is
        running&mdash;its parameters are repository URLs, and it does
        all its work through Subversion's Repository Access (RA)
        interfaces.  All it requires is read access to the source
        repository and read/write access to the destination
        repository.</para>

      <note>
        <para>When using <command>svnsync</command> against a remote
          source repository, the Subversion server for that repository
          must be running Subversion version 1.4 or later.</para>
      </note>

      <para>Assuming you already have a source repository that you'd
        like to mirror, the next thing you need is an empty target
        repository that will actually serve as that mirror.  This
        target repository can use either of the available filesystem
        data-store backends (see <xref
        linkend="svn.reposadmin.basics.backends" />), but it must not
        yet have any version history in it.  The protocol that
        <command>svnsync</command> uses to communicate revision information
        is highly sensitive to mismatches between the versioned
        histories contained in the source and target repositories.
        For this reason, while <command>svnsync</command> cannot
        <emphasis>demand</emphasis> that the target repository be
        read-only,
        <footnote>
          <para>In fact, it can't truly be read-only, or
            <command>svnsync</command> itself would have a tough time
            copying revision history into it.</para>
        </footnote>
        allowing the revision history in the target repository to
        change by any mechanism other than the mirroring process is a
        recipe for disaster.</para>

      <warning>
        <para>Do <emphasis>not</emphasis> modify a mirror repository
          in such a way as to cause its version history to deviate
          from that of the repository it mirrors.  The only commits
          and revision property modifications that ever occur on that
          mirror repository should be those performed by the
          <command>svnsync</command> tool.</para>
      </warning>

      <para>Another requirement of the target repository is that the
        <command>svnsync</command> process be allowed to modify
        revision properties.  Because <command>svnsync</command> works
        within the framework of that repository's hook system, the
        default state of the repository (which is to disallow revision
        property changes; see <xref
        linkend="svn.ref.reposhooks.pre-revprop-change" />) is
        insufficient.  You'll need to explicitly implement the
        pre-revprop-change hook, and your script must allow
        <command>svnsync</command> to set and change revision
        properties.  With those provisions in place, you are ready to
        start mirroring repository revisions.</para>

      <tip>
        <para>It's a good idea to implement authorization measures
          that allow your repository replication process to perform
          its tasks while preventing other users from modifying the
          contents of your mirror repository at all.</para>
      </tip>

      <para>Let's walk through the use of <command>svnsync</command>
        in a somewhat typical mirroring scenario.  We'll pepper this
        discourse with practical recommendations, which you are free to
        disregard if they aren't required by or suitable for your
        environment.</para>

      <para>As a service to the fine developers of our favorite
        version control system, we will be mirroring the public
        Subversion source code repository and exposing that mirror
        publicly on the Internet, hosted on a different machine than
        the one on which the original Subversion source code
        repository lives.  This remote host has a global configuration
        that permits anonymous users to read the contents of
        repositories on the host, but requires users to authenticate
        to modify those repositories.  (Please forgive us for
        glossing over the details of Subversion server configuration
        for the moment&mdash;those are covered thoroughly in <xref
        linkend="svn.serverconfig" />.)  And for no other reason than
        that it makes for a more interesting example, we'll be driving
        the replication process from a third machine&mdash;the one that
        we currently find ourselves using.</para>

      <para>First, we'll create the repository which will be our
        mirror.  This and the next couple of steps do require shell
        access to the machine on which the mirror repository will
        live.  Once the repository is all configured, though, we
        shouldn't need to touch it directly again.</para>

      <screen>
$ ssh admin@svn.example.com \
      "svnadmin create /var/svn/svn-mirror"
admin@svn.example.com's password: ********
$
</screen>

      <para>At this point, we have our repository, and due to our
        server's configuration, that repository is now
        <quote>live</quote> on the Internet.  Now, because we don't
        want anything modifying the repository except our replication
        process, we need a way to distinguish that process from other
        would-be committers.  To do so, we use a dedicated username
        for our process.  Only commits and revision property
        modifications performed by the special username
        <literal>syncuser</literal> will be allowed.</para>

      <para>We'll use the repository's hook system both to allow the
        replication process to do what it needs to do and to enforce
        that only it is doing those things.  We accomplish this by
        implementing two of the repository event
        hooks&mdash;pre-revprop-change and start-commit.  Our
        <filename>pre-revprop-change</filename> hook script is found
        in <xref
        linkend="svn.reposadmin.maint.replication.pre-revprop-change"
        />, and basically verifies that the user attempting the
        property changes is our <literal>syncuser</literal> user.  If
        so, the change is allowed; otherwise, it is denied.</para>

      <example id="svn.reposadmin.maint.replication.pre-revprop-change">
        <title>Mirror repository's pre-revprop-change hook script</title>

        <programlisting>
#!/bin/sh 

USER="$3"

if [ "$USER" = "syncuser" ]; then exit 0; fi

echo "Only the syncuser user may change revision properties" &gt;&amp;2
exit 1
</programlisting>
      </example>


      <para>That covers revision property changes.  Now we need to
        ensure that only the <literal>syncuser</literal> user is
        permitted to commit new revisions to the repository.  We do
        this using a <filename>start-commit</filename> hook scripts
        such as the one in <xref
        linkend="svn.reposadmin.maint.replication.start-commit"
        />.</para>

      <example id="svn.reposadmin.maint.replication.start-commit">
        <title>Mirror repository's start-commit hook script</title>

        <programlisting>
#!/bin/sh 

USER="$2"

if [ "$USER" = "syncuser" ]; then exit 0; fi

echo "Only the syncuser user may commit new revisions" &gt;&amp;2
exit 1
</programlisting>
      </example>

      <para>After installing our hook scripts and ensuring that they
        are executable by the Subversion server, we're finished with
        the setup of the mirror repository.  Now, we get to actually
        do the mirroring.</para>

      <para>The first thing we need to do with
        <command>svnsync</command> is to register in our target
        repository the fact that it will be a mirror of the source
        repository.  We do this using the <command>svnsync
        initialize</command> subcommand.  The URLs we provide point to
        the root directories of the target and source repositories,
        respectively.  In Subversion 1.4, this is required&mdash;only
        full mirroring of repositories is permitted.  In Subversion
        1.5, though, you can use <command>svnsync</command> to mirror
        only some subtree of the repository, too.</para>

      <screen>
$ svnsync help init
initialize (init): usage: svnsync initialize DEST_URL SOURCE_URL

Initialize a destination repository for synchronization from
another repository.
&hellip;
$ svnsync initialize http://svn.example.com/svn-mirror \
                     http://svn.collab.net/repos/svn \
                     --sync-username syncuser --sync-password syncpass
Copied properties for revision 0.
$
</screen>

      <para>Our target repository will now remember that it is a
        mirror of the public Subversion source code repository.
        Notice that we provided a username and password as arguments
        to <command>svnsync</command>&mdash;that was required by the
        pre-revprop-change hook on our mirror repository.</para>

      <note>
        <para>In Subversion 1.4, the values given to
          <command>svnsync</command>'s <option>--username</option> and
          <option>--password</option> command-line options were used
          for authentication against both the source and destination
          repositories.  This caused problems when a user's
          credentials weren't exactly the same for both repositories,
          especially when running in noninteractive mode (with the
          <option>--non-interac