Docker and osquery

If you are into security you might have heard about osquery. It is extremely powerful tool that can be used for various purposes:

  • Real time endpoint monitoring
  • Anomaly detection
  • File integrity monitoring
  • Metrics (prometheus)
  • Container (docker) monitoring
  • syslog aggregation

Numerous enterprises big and small from all verticals are using it, or planning on using it. It is being deployed to production servers as well as employee desktop/laptops. It has first class support for various flavors of Linux and macOS. Windows functionality is maturing, thanks to open source community contributions and Facebook’s efforts.

Recently we (Uptycs) started publishing docker images with latest osquery version. We published images for various versions of Ubuntu and CentOS. Ideally running osquery in docker container doesn’t make sense unless you are using CoreOS Container Linux. But if you are just playing with osquery and want to test some functionality docker images are ideal. osquery comes with two binaries. Interactive shell osqueryi and osqueryd daemon.

Interactive shell can be launched as follows. It will present a SQL prompt:

$ docker run -it uptycs/osquery:2.7.0 osqueri
Using a virtual database. Need help, type '.help'

You can run sample queries like:

osquery> SELECT * FROM processes;

When running inside the container osquery will only return information available to it from within the container. In the case of processes which are retrieved from /proc osquery will return the processes running inside the container. In this case it will be one process: /usr/bin/osqueryi

If you omit the command part it will launch the osquery daemon with a warning:

$ docker run -it uptycs/osquery:2.7.0
W0919 19:46:14.058230 7 init.cpp:649] Error reading config: config file does not exist: /etc/osquery/osquery.conf

This is because the Dockerfile is configured to run the osquery daemon by default with the following arguments:

osqueryd --flagfile /etc/osquery/osquery.flags --config_path /etc/osquery/osquery.conf

Flags file and the config file are meant to be provided from the host. osquery have extensive number of command line flags. Optionally configuration file can also be provided to the container. Refer to osquery configuration documentation on what can be specified in conf file.

Assuming osquery.flags and osquery.conf are created on host machine in directory /some/path, osquery daemon can be launched as:

$ docker run -it -v /some/path:/etc/osquery uptycs/osquery:2.7.0

osquery can also gather information about docker . When osquery is running inside container, it cannot talk to the docker daemon running on the host machine. If you expose the document UNIX domain socket from host to the container osquery can query/gather information about docker images, containers, volumes, networks, labels etc

$ docker run -it -v /some/path:/etc/osquery -v/var/run/docker.sock:/var/run/docker.sock uptycs/osquery:2.7.0

If logging is configured, osquery daemon needs to identify itself to the log endpoint.
–host_identifier flag should be appropriately configured. If hostname value is used for host identifier, you might want to start docker with hostname option:

$ docker run -it --hostname -v /some/path:/etc/osquery -v/var/run/docker.sock:/var/run/docker.sock uptycs/osquery:2.7.0

host_identifier with uuid value is not appropriate if you are planning on launching multiple osquery docker instances or if you have osquery running on the host also. Docker containers will end up sharing the same UUID.

As I mentioned above osquery in docker only makes sense for playing with osquery or testing it. In real deployments it should be running on the host or virtual machine. In case of CoreOS Container Linux there is no easy way to run any service on the host/virtual machine. On Container Linux osquery can be run inside toolbox which uses systemd-nspawn.  I will cover this in separate topic.

Docker and osquery