> ## Documentation Index
> Fetch the complete documentation index at: https://watchdog.sdslabs.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Watchdog Internals

> CLI usage, what Watchdog tracks, and how it works under the hood.

As mentioned earlier, Watchdog is a Rust binary installed on the server which communicates with the Keyhouse repository to check for privileges, and notifies of any login attempts, successful or otherwise, through Slack messages. As an added feature, we also notify you about any administrative activities like `sudo`.

It works by plugging callbacks into the commands `sudo`, `su`, and `ssh` using [Pluggable Authentication Modules](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/managing_smart_cards/pluggable_authentication_modules) (PAM) (to send messages to Slack) and the [SSH Authorized Keys Command](https://scalesec.com/blog/just-in-time-ssh-provisioning/) (to authenticate with Keyhouse).

## Watchdog CLI Usage

The `watchdog` binary is located at:

```
/opt/watchdog/bin
```

To run any command, navigate to that folder and use `sudo ./watchdog <command>`.

### 1. View Logs

```bash theme={null}
sudo ./watchdog logs all --level info
```

* View logs for all parts of Watchdog (`ssh`, `sudo`, `su`, etc.).
* You can replace `info` with `warn` or `error` to filter severity.
* This helps debug or audit what Watchdog is doing.

### 2. Update Users and Groups

```bash theme={null}
sudo ./watchdog update
```

* Syncs users and groups from the Keyhouse system into your server, by fetching and applying the diff since the last sync (see [Keyhouse Sync](https://watchdog.sdslabs.co/how-it-works/keyhouse-sync.md) for how this works).
* Needs something to trigger it: CI, cron, or the `auto-update` build feature (which runs a full sync on every SSH login). See [Setup](https://watchdog.sdslabs.co/setup.md#keeping-watchdog-in-sync-with-keyhouse).
* Requires `base_commit.txt` from a previous sync; on a fresh install, run `update --all` first.
* Pass `--all` to force a full resync instead of an incremental diff:

```bash theme={null}
sudo ./watchdog update --all
```

### 3. Help

You can always view available commands and flags by running:

```bash theme={null}
sudo ./watchdog help
```

Example output:

```
Watchdog 0.1.0
SDSLabs <contact@sdslabs.co>
Update users and groups from Keyhouse

USAGE:
    watchdog [SUBCOMMAND]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

SUBCOMMANDS:
    auth      Authorizes users based on from keyhouse repository. This command is passed through
              `AuthorizedKeysCommand` in sshd_config.
    config    Get or set Watchdog configuration
    help      Prints this message or the help of the given subcommand(s)
    logs      Fetch logs from watchdog components
    ssh       Handles the PAM SSH calls by pam_exec for Watchdog
    su        Handles the PAM su calls by pam_exec for Watchdog
    sudo      Handles the PAM sudo calls by pam_exec for Watchdog
    update
```

`auth`, `ssh`, `su`, and `sudo` are called by sshd and PAM; you won't normally run them yourself.

### Optional: Add to `$PATH`

To use `watchdog` globally without navigating every time (the [Setup](https://watchdog.sdslabs.co/setup.md) guide already does this):

```bash theme={null}
sudo ln -s /opt/watchdog/bin/watchdog /usr/bin/watchdog
```

Now you can simply do:

```bash theme={null}
sudo watchdog logs all --level info
```

## What Does Watchdog Track?

### 1. Auth Check (`handle_auth`)

* When someone tries to log in with an SSH key, Watchdog checks if the key is valid.
* If it is, everything goes smoothly.
* If not, it sends an alert saying, "user tried to log in on user\@server"

![Auth check Slack alert](https://mintcdn.com/watchdog-docs/jan3tIWKPXZCjmik/images/Lbike9zRCmv0P7h4i4Li.png?fit=max\&auto=format\&n=jan3tIWKPXZCjmik\&q=85\&s=345e7d1a1a84ea76b4d82b1f1a2db796)

### 2. SSH Session (`handle_ssh`)

* When a user starts an SSH session (remote login), Watchdog notices.
* It logs the event and sends a simple report: "User X logged in on user\@server"

![SSH session Slack alert](https://mintcdn.com/watchdog-docs/jan3tIWKPXZCjmik/images/qdZ9SsgwegkaeTzws3qR.png?fit=max\&auto=format\&n=jan3tIWKPXZCjmik\&q=85\&s=f86014e33a690b91740864564180a780)

### 3. Sudo Use (`handle_sudo`)

* When someone uses `sudo` to run admin commands, Watchdog tracks it.
* It logs the user's name and the folder they were in, and also, the command they used, in a thread.
* Then it sends a message: "user attempted sudo on server".

![Sudo use Slack alert](https://mintcdn.com/watchdog-docs/jan3tIWKPXZCjmik/images/002ZtmxvKSVtEgP9Mad0.png?fit=max\&auto=format\&n=jan3tIWKPXZCjmik\&q=85\&s=4cc2f163e76e3d7e65285def52b9f3f9)

### 4. Su Command (`handle_su`)

* When a user switches to another user using `su`, Watchdog catches that too.
* It tracks who switched to whom.
* Then it sends a simple message: "User A switched to user B."

![Su command Slack alert](https://mintcdn.com/watchdog-docs/jan3tIWKPXZCjmik/images/NTKepUATDDfl0Mk2Mi8Q.png?fit=max\&auto=format\&n=jan3tIWKPXZCjmik\&q=85\&s=4fe247898d863e0d5912f4e84ce878e7)

![Su command Slack alert thread](https://mintcdn.com/watchdog-docs/jan3tIWKPXZCjmik/images/9pF4RgIjdAn39Su6Q1J3.png?fit=max\&auto=format\&n=jan3tIWKPXZCjmik\&q=85\&s=ba70d0a4bef9ed5fa1de7915509bfbc9)

## How does it work?

* It uses environment variables like `PAM_USER`, `PAM_TYPE`, and `PAM_RUSER` to know who did what.
* It reads some settings from a config file.
* It uses a notifier to send simple messages (like alerts).
* It uses forking to run some tasks in the background without blocking other processes.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.