Raspberry Pi Data Logger with InfluxDB and Grafana

November 3, 2016

A need popped up at work for a data logger for various lab tasks. Quickly looking at the market, I failed to identify a lab tool for data logging (cheap, easy but powerful setup, remote access); something for researchers and scientists. I decided a Raspberry Pi with some input buffering would be ideal for the task. This is my roll your own data logger, put together on Saturday – showing what is possible quickly and potential with more development time.

Hardware Setup

  • Raspberry Pi (3) – I used a 3 for integrated wifi and speed but any Pi should work.
  • SD Card – A reliable brand as the database will be writing frequently. I used a 16GB SanDisk Extreme
  • Some form of IO buffer or input device. I first developed with a DHT22 temperature/humidity sensor, followed by a potential divider/zener diode digital logic buffer then finally a Pimoroni Automation HAT for analogue voltage. I plan on creating a custom multi-channel buffered 24V logic and 0-10V analogue input card to partner with this, for industrial device testing.

Install InfluxDB and Grafana on Raspberry Pi

A while back I attempted this but ran into problems installing and configuring InfluxDB and Grafana. I had attempted to user Docker but without success. Thankfully, .deb packages now exist in the Debian ‘Stretch’ repository for armhf (Raspberry Pi). You and either add ‘Stretch to your list of sources or download the packages with wget:

# update these URLs by looking at the website: https://packages.debian.org/sid/grafana
sudo apt-get update
sudo apt-get upgrade
wget http://ftp.us.debian.org/debian/pool/main/i/influxdb/influxdb_1.0.2+dfsg1-1_armhf.deb
sudo dpkg -i influxdb_1.0.2+dfsg1-1_armhf.deb
wget http://ftp.us.debian.org/debian/pool/main/g/grafana/grafana-data_2.6.0+dfsg-3_all.deb # grafana data is a dependancy for grafana
sudo dpkg -i grafana-data_2.6.0+dfsg-3_all.deb
sudo apt-get install -f
wget http://ftp.us.debian.org/debian/pool/main/g/grafana/grafana_2.6.0+dfsg-3_armhf.deb
sudo dpkg -i grafana_2.6.0+dfsg-3_armhf.deb
sudo apt-get install -f

Setup InfluxDB

Installed via the .deb package, InfluxDB creates a service and enables at startup so there is little more configuration to do. One must create the databases that our scripts will be saving into. This can be done via the web gui. Navigate to your Pi IP on port 8083 (default InfluxDB admin port – http://localhost:8083 if you’re doing this on the Pi). Using the ‘Query Templates’ dropdown, one can create a database with ‘Create Database’ – it’s fairly self explainatory. Make one called ‘logger’ to store our sample data.

InfluxDB Create Database

Create Logging Script

The example below uses Pimoroni’s Automation HAT but can be easily adapted to a GPIO pin or DHT22 using the commented lines. It’s worth having a read of the InfluxDB Key Concepts but essentially, I create a unique tag for each time the script runs called ‘run’. This enables a user to differentiate between test setups with no backend jiggery pockery when we come to Grafana. The measurement name can be set via the command arguments, to enable a user to set different testing sessions, currently by default it is ‘dev’. You’ll need the InfluxDB Python module for this script, which you can install like this:

sudo pip install influxdb

import time
import sys
import datetime
from influxdb import InfluxDBClient
import automationhat
#import RPi.GPIO as GPIO

# Set this variables, influxDB should be localhost on Pi
host = "localhost"
port = 8086
user = "root"
password = "root"

# The database we created
dbname = "logger"
# Sample period (s)
interval = 1

# For GPIO
# channel = 14¬
# GPIO.setmode(GPIO.BCM)¬
# GPIO.setup(channel, GPIO.IN)¬

# Allow user to set session and runno via args otherwise auto-generate
if len(sys.argv) > 1:
    if (len(sys.argv) < 3):
        print "Must define session and runNo!!"
        sys.exit()
    else:
        session = sys.argv[1]
        runNo = sys.argv[2]
else:
    session = "dev"
    now = datetime.datetime.now()
    runNo = now.strftime("%Y%m%d%H%M")

print "Session: ", session
print "runNo: ", runNo

# Create the InfluxDB object
client = InfluxDBClient(host, port, user, password, dbname)

# Run until keyboard out
try:
    while True:
        # This gets a dict of the three values
        vsense = automationhat.analog.read()
        op = automationhat.input.read()
        # gpio = GPIO.input(channel)
        print vsense
        print op
        iso = time.ctime()

        json_body = [
        {
          "measurement": session,
              "tags": {
                  "run": runNo,
                  },
              "time": iso,
              "fields": {
                  "op1" : op['one'], "op2" : op['two'], "op3" : op['three'],
                  "vsense1" : vsense['one'],"vsense2" : vsense['two'],"vsense3" : vsense['three']
                  # ,"gpio" : gpio
              }
          }
        ]

        # Write JSON to InfluxDB
        client.write_points(json_body)
        # Wait for next sample
        time.sleep(interval)

except KeyboardInterrupt:
    pass

Save the script as ‘logger.py’ and run using python logger.py. To run the script in the background and indefinately as a ssh user use nohup python logger.py &.

Setup Grafana

Unlike InfluxDB, Grafana doesn’t enable it’s service, so do this to enable at boot and start the service now:

sudo systemctl enable grafana-server
sudo systemctl start grafana-server

Navigating to port 3000, you should be presented with the Grafana web GUI. Grafana has plenty of powerful querying tools to make lots of pretty and informative graphs. Have a read of the Getting Started, and in particular the InfluxDB section.

The first thing you will need to do is add the ‘logger’ database as a Datasource. Navigate to Datasource->Add New and fill in as below

Create the 'logger' InfluxDB database as a Datasource
Create the ‘logger’ InfluxDB database as a Datasource

After this, you need to setup a Dashboard. Click the top dropdown button, press ‘New’ and create a dashboard called ‘logger’. The dashboards consist of rows. Clicking on the green bar on the left-hand edge provides a dropdown where one can add graphs etc – again, the Getting Started explains this better than me. Below are a couple of captures of how I setup the Automation HAT.

Log Away

With this short introduction you should have the framework for a basic logging device and see the potential for real-world analytics of almost anything! I’m quite excited using the combination of a time based database like InfluxDB and powerful web graphing package Grafana.