phabricator/src/docs/user/userguide/diffusion_hosting.diviner

@title Diffusion User Guide: Repository Hosting
@group userguide

Guide to configuring Phabricator repository hosting.

= Overview =

Phabricator can host repositories and provide authenticated read and write
access to them over HTTP and SSH. This document describes how to configure
repository hosting.

= Understanding Supported Protocols =

Phabricator supports hosting over these protocols:

| VCS | SSH | HTTP |
|-----|-----|------|
| Git | Supported | Supported |
| Mercurial | Supported | Supported |
| Subversion | Supported | Not Supported |

All supported protocols handle reads (pull/checkout/clone) and writes
(push/commit). Of the two protocols, SSH is generally more robust, secure and
performant, but HTTP is easier to set up and supports anonymous access.

| | SSH | HTTP |
| |-----|------|
| Reads | Yes | Yes |
| Writes | Yes | Yes |
| Authenticated Access | Yes | Yes |
| Push Logs | Yes | Yes |
| Commit Hooks | Yes | Yes |
| Anonymous Access | No | Yes |
| Security | Better (Asymmetric Key) | Okay (Password) |
| Performance | Better | Okay |
| Setup | Hard | Easy |

Each repository can be configured individually, and you can use either protocol,
or both, or a mixture across different repositories.

SSH is recommended unless you need anonymous access, or are not able to
configure it for technical reasons.

= Configuring System User Accounts =

Phabricator uses as many as three user accounts. This section will guide you
through creating and configuring them. These are system user accounts on the
machine Phabricator runs on, not Phabricator user accounts.

The system accounts are:

  - The user the daemons run as. We'll call this `daemon-user`. For more
    information on the daemons, see @{article:Managing Daemons with phd}. This
    user is the only user which will interact with the repositories directly.
    Other accounts will `sudo` to this account in order to perform VCS
    operations.
  - The user the webserver runs as. We'll call this `www-user`. If you do not
    plan to make repositories available over HTTP, you do not need to perform
    any special configuration for this user.
  - The user that users will connect over SSH as. We'll call this `vcs-user`.
    If you do not plan to make repositories available over SSH, you do not need
    to perform any special configuration for this user.

To configure these users:

  - Create a `daemon-user` if one does not already exist (you can call this user
    whatever you want, or use an existing account). When you start the daemons,
    start them using this user.
  - Create a `www-user` if one does not already exist. Run your webserver as
    this user. In most cases, this user will already exist.
  - Create a `vcs-user` if one does not already exist. Common names for this
    user are `git` or `hg`. When users clone repositories, they will use a URI
    like `vcs-user@phabricator.yourcompany.com`.

Now, allow the `vcs-user` and `www-user` to `sudo` as the `daemon-user`. Add
this to `/etc/sudoers`, using `visudo` or `sudoedit`.

If you plan to use SSH:

  vcs-user ALL=(daemon-user) SETENV: NOPASSWD: /path/to/bin/git-upload-pack, /path/to/bin/git-receive-pack, /path/to/bin/hg, /path/to/bin/svnserve

If you plan to use HTTP:

  www-user ALL=(daemon-user) SETENV: NOPASSWD: /usr/bin/git-http-backend, /usr/bin/hg

Replace `vcs-user`, `www-user` and `daemon-user` with the right usernames for
your configuration. Make sure all the paths point to the real locations of the
binaries on your system. You can omit any binaries associated with VCSes you do
not use.

Adding these commands to `sudoers` will allow the daemon and webserver users to
write to repositories as the daemon user.

Before saving and closing `/etc/sudoers`, look for this line:

  Defaults    requiretty

If it's present, comment it out by putting a `#` at the beginning of the line.
With this option enabled, VCS SSH sessions won't be able to use `sudo`.

If you're planning to use SSH, you should also edit `/etc/passwd` and
`/etc/shadow` to make sure the `vcs-user` account is set up correctly.

  - Open `/etc/shadow` and find the line for the `vcs-user` account.
    - The second field (which is the password field) must not be set to
      `!!`. This value will prevent login. If it is set to `!!`, edit it
      and set it to `NP` ("no password") instead.
  - Open `/etc/passwd` and find the line for the `vcs-user` account.
    - The last field (which is the login shell) must be set to a real shell.
      If it is set to something like `/bin/false`, then `sshd` will not be able
      to execute commands. Instead, you should set it to a real shell, like
      `/bin/sh`.

Finally, once you've configured `/etc/sudoers`, `/etc/shadow` and `/etc/passwd`,
set `phd.user` to the `daemon-user`:

  phabricator/ $ ./bin/config set phd.user daemon-user

If you're using a `vcs-user`, you should also configure that here:

  phabricator/ $ ./bin/config set diffusion.ssh-user vcs-user

= Configuring HTTP =

If you plan to use authenticated HTTP, you need to set
`diffusion.allow-http-auth` in Config. If you don't plan to use HTTP, or plan to
use only anonymous HTTP, you can leave this setting disabled.

If you plan to use authenticated HTTP, you'll also need to configure a VCS
password in {nav Settings > VCS Password}.

Your VCS password must be a different password than your main Phabricator
password because VCS passwords are very easy to accidentally disclose. They are
often stored in plaintext in world-readable files, observable in `ps` output,
and present in command output and logs. We strongly encourage you to use SSH
instead of HTTP to authenticate access to repositories.

Otherwise, if you've configured system accounts above, you're all set. No
additional server configuration is required to make HTTP work.

= Configuring SSH =

SSH access requires some additional setup. Here's an overview of how setup
works:

  - You'll move the normal `sshd` daemon to another port, like `222`. When
    connecting to the machine to administrate it, you'll use this alternate
    port to get a normal login shell.
  - You'll run a highly restricted `sshd` on port 22, with a special locked-down
    configuration that uses Phabricator to authorize users and execute commands.
  - The `sshd` on port 22 **MUST** be 6.2 or newer, because Phabricator relies
    on the `AuthorizedKeysCommand` option.

Here's a walkthrough of how to perform this configuration in detail:

**Move Normal SSHD**: Be careful when editing the configuration for `sshd`. If
you get it wrong, you may lock yourself out of the machine. Restarting `sshd`
generally will not interrupt existing connections, but you should exercise
caution. Two strategies you can use to mitigate this risk are: smoke-test
configuration by starting a second `sshd`; and use a `screen` session which
automatically repairs configuration unless stopped.

To smoke-test a configuration, just start another `sshd` using the `-f` flag:

  sudo /path/to/sshd -f /path/to/config_file.edited

You can then connect and make sure the edited config file is valid before
replacing your primary configuration file.

To automatically repair configuration, start a `screen` session with a command
like this in it:

  sleep 60 ; mv sshd_config.good sshd_config ; /etc/init.d/sshd restart

The specific command may vary for your system, but the general idea is to have
the machine automatically restore configuration after some period of time if
you don't stop it. If you lock yourself out, this will fix things automatically.

Now that you're ready to edit your configuration, open up your `sshd` config
(often `/etc/ssh/sshd_config`) and change the `Port` setting to some other port,
like `222` (you can choose any port other than 22).

  Port 222

Very carefully, restart `sshd`. Verify that you can connect on the new port:

  ssh -p 222 ...

**Configure and Start Phabricator SSHD**: Now, configure and start a second
`sshd` instance which will run on port `22`. This instance will use a special
locked-down configuration that uses Phabricator to handle authentication and
command execution.

There are three major steps:

  - Create a `phabricator-ssh-hook.sh` file.
  - Create a `sshd_phabricator` config file.
  - Start a copy of `sshd` using the new configuration.

**Create `phabricator-ssh-hook.sh`**: Copy the template in
`phabricator/resources/sshd/phabricator-ssh-hook.sh` to somewhere like
`/usr/libexec/phabricator-ssh-hook.sh` and edit it to have the correct
settings.

Both the script itself **and** the parent directory the script resides in must
be owned by `root`, and the script must have `755` permissions:

```
$ sudo chown root /path/to/somewhere/
$ sudo chown root /path/to/somewhere/phabricator-ssh-hook.sh
$ sudo chmod 755 /path/to/somewhere/phabricator-ssh-hook.sh
```

If you don't do this, `sshd` will refuse to execute the hook.

**Create `sshd_config` for Phabricator**: Copy the template in
`phabricator/resources/sshd/sshd_config.phabricator.example` to somewhere like
`/etc/ssh/sshd_config.phabricator`.

Open the file and edit the `AuthorizedKeysCommand`,
`AuthorizedKeysCommandUser`, and `AllowUsers` settings to be correct for your
system.

**Start SSHD**: Now, start the Phabricator `sshd`:

  sudo /path/to/sshd -f /path/to/sshd_config.phabricator

If you did everything correctly, you should be able to run this:

  echo {} | ssh vcs-user@phabricator.yourcompany.com conduit conduit.ping

...and get a response like this:

  {"result":"orbital","error_code":null,"error_info":null}

(If you get an authentication error, make sure you added your public key in
**Settings > SSH Public Keys**.) If you're having trouble, check the
troubleshooting section below.

= Authentication Over HTTP =

To authenticate over HTTP, users should configure a **VCS Password** in the
**Settings** screen. This panel is available only if `diffusion.allow-http-auth`
is enabled.

= Authentication Over SSH =

To authenticate over SSH, users should add **SSH Public Keys** in the
**Settings** screen.

= Cloning a Repository =

If you've already set up a hosted repository, you can try cloning it now. To
do this, browse to the repository's main screen in Diffusion. You should see
clone commands at the top of the page.

To clone the repository, just run the appropriate command.

If you don't see the commands or running them doesn't work, see below for tips
on troubleshooting.

= Troubleshooting HTTP =

Some general tips for troubleshooting problems with HTTP:

  - Make sure `diffusion.allow-http-auth` is enabled in your Phabricator config.
  - Make sure HTTP serving is enabled for the repository you're trying to clone.
    You can find this in {nav Edit Repository > Hosting}.
  - Make sure you've configured a VCS password. This is separate from your main
    account password. You can configure this in {nav Settings > VCS Password}.
  - Make sure the main repository screen in Diffusion shows a clone/checkout
    command for HTTP. If it doesn't, something above isn't set up correctly:
    double-check your configuration. You should see a `svn checkout http://...`,
    `git clone http://...` or `hg clone http://...` command. Run that command
    verbatim to clone the repository.

If you're using Git, using `GIT_CURL_VERBOSE` may help assess login failures.
To do so, specify it on the command line before the `git clone` command, like
this:

  $ GIT_CURL_VERBOSE=1 git clone ...

This will make `git` print out a lot more information. Particularly, the line
with the HTTP response is likely to be useful:

  < HTTP/1.1 403 Invalid credentials.

In many cases, this can give you more information about what's wrong.

= Troubleshooting SSH =

Some general tips for troubleshooting problems with SSH:

  - Check that you've configured `diffusion.ssh-user`.
  - Check that you've configured `phd.user`.
  - Make sure SSH serving is enabled for the repository you're trying to clone.
    You can change this setting from a main repository screen in Diffusion by
    {nav Edit Repository >
    Edit Hosting >
    Host Repository on Phabricator >
    Save and Continue >
    SSH Read Only or Read/Write >
    Save Changes}.
  - Make sure you've added an SSH public key to your account. You can do this
    in {nav Settings > SSH Public Keys}.
  - Make sure the main repository screen in Diffusion shows a clone/checkout
    command for SSH. If it doesn't, something above isn't set up correctly.
    You should see an `svn checkout svn+ssh://...`, `git clone ssh://...` or
    `hg clone ssh://...` command. Run that command verbatim to clone the
    repository.
  - Check your `phabricator-ssh-hook.sh` file for proper settings.
  - Check your `sshd_config.phabricator` file for proper settings.

To troubleshoot SSH setup: connect to the server with `ssh`, without running
a command. You may need to use the `-T` flag. You should see a message like
this one:

  $ ssh -T dweller@secure.phabricator.com
  phabricator-ssh-exec: Welcome to Phabricator.

  You are logged in as alincoln.

  You haven't specified a command to run. This means you're requesting an
  interactive shell, but Phabricator does not provide an interactive shell over
  SSH.

  Usually, you should run a command like `git clone` or `hg push` rather than
  connecting directly with SSH.

  Supported commands are: conduit, git-receive-pack, git-upload-pack, hg,
  svnserve.

If you see this message, all your SSH stuff is configured correctly. **If you
get a login shell instead, you've missed some major setup step: review the
documentation above.** If you get some other sort of error, double check these
settings:

  - You're connecting as the `vcs-user`.
  - The `vcs-user` has `NP` in `/etc/shadow`.
  - The `vcs-user` has `/bin/sh` or some other valid shell in `/etc/passwd`.
  - Your SSH key is correct, and you've added it to Phabricator in the Settings
    panel.

If you can get this far, but can't execute VCS commands like `git clone`, there
is probably an issue with your `sudoers` configuration. Check:

  - Your `sudoers` file is set up as instructed above.
  - You've commented out `Defaults requiretty` in `sudoers`.
  - You don't have multiple copies of the VCS binaries (like `git-upload-pack`)
    on your system. You may have granted sudo access to one, while the VCS user
    is trying to run a different one.
  - You've configured `phd.user`.
  - The `phd.user` has read and write access to the repositories.

It may also be helpful to run `sshd` in debug mode:

  $ /path/to/sshd -d -d -d -f /path/to/sshd_config.phabricator

This will run it in the foreground and emit a large amount of debugging
information.

Finally, you can usually test that `sudoers` is configured correctly by
doing something like this:

  $ su vcs-user
  $ sudo -E -n -u daemon-user -- /path/to/some/vcs-binary --help

That will try to run the binary via `sudo` in a manner similar to the way that
Phabricator will run it. This can give you better error messages about issues
with `sudoers` configuration.

= Miscellaneous Troubleshooting =

  - If you're getting an error about `svnlook` not being found, add the path
    where `svnlook` is located to the Phabricator configuration
    `environment.append-paths` (even if it already appears in PATH). This issue
    is caused by SVN wiping the environment (including PATH) when invoking
    commit hooks.

No Direct Pushes
================

You may get an error about "No Direct Pushes" when trying to push. This means
you are pushing directly to the repository instead of pushing through
Phabricator. This is not supported: writes to hosted repositories must go
through Phabricator so it can perform authentication, enforce permissions,
write logs, proxy requests, apply rewriting, etc.

One way to do a direct push by mistake is to use a `file:///` URI to interact
with the repository from the same machine. This is not supported. Instead, use
one of the repository URIs provided in the web interface, even if you're
working on the same machine.

Another way to do a direct push is to misconfigure SSH (or not configure it at
all) so that none of the logic described above runs and you just connect
normally as a system user. In this case, the `ssh` test described above will
fail (you'll get a command prompt when you connect, instead of the message you
are supposed to get, as described above).

If you encounter this error: make sure you're using a remote URI given to
you by Diffusion in the web interface, then run through the troubleshooting
steps above carefully.

Sometimes users encounter this problem because they skip this whole document
assuming they don't need to configure anything. This will not work, and you
MUST configure things as described above for hosted repositories to work.

The technical reason this error occurs is that the `PHABRICATOR_USER` variable
is not defined in the environment when commit hooks run. This variable is set
by Phabricator when a request passes through the authentication layer that this
document provides instructions for configuring. Its absence indicates that the
request did not pass through Phabricator.


= Next Steps =

Once hosted repositories are set up:

  - learn about commit hooks with @{article:Diffusion User Guide: Commit Hooks}.