User Guide

Authentication

The EMQ broker supports to authenticate MQTT clients with ClientID, Username/Password, IpAddress and even HTTP Cookies.

The authentication is provided by a list of plugins such as MySQL, PostgreSQL and Redis…

If we enable several authentication plugins at the same time, the authentication process:

           ----------------           ----------------           -------------
Client --> |   Username   | -ignore-> |   ClientID   | -ignore-> | Anonymous |
           ----------------           ----------------           -------------
                  |                         |                         |
                 \|/                       \|/                       \|/
            allow | deny              allow | deny              allow | deny

The authentication plugins implemented by default:

Plugin Description
emq_auth_clientid ClientId Auth Plugin
emq_auth_username Username Auth Plugin
emq_auth_ldap LDAP Auth Plugin
emq_auth_http HTTP Auth/ACL Plugin
emq_auth_mysql MySQL Auth/ACL Plugin
emq_auth_pgsql Postgre Auth/ACL Plugin
emq_auth_redis Redis Auth/ACL Plugin
emq_auth_mongo MongoDB Auth/ACL Plugin

Allow Anonymous

Configure etc/emq.conf to allow anonymous authentication:

## Allow Anonymous authentication
mqtt.allow_anonymous = true

Username/Password

Authenticate MQTT client with Username/Password:

Configure default users in etc/plugins/emq_auth_username.conf:

auth.user.$N.username = admin
auth.user.$N.password = public

Enable emq_auth_username plugin:

./bin/emqttd_ctl plugins load emq_auth_username

Add user by ‘./bin/emqttd_ctl users’ command:

$ ./bin/emqttd_ctl users add <Username> <Password>

ClientId

Authentication with MQTT ClientId.

Configure Client Ids in etc/plugins/emq_auth_clientid.conf:

auth.client.$N.clientid = clientid
auth.client.$N.password = passwd

Enable emq_auth_clientid plugin:

./bin/emqttd_ctl plugins load emq_auth_clientid

LDAP

etc/plugins/emq_auth_ldap.conf:

auth.ldap.servers = 127.0.0.1

auth.ldap.port = 389

auth.ldap.timeout = 30

auth.ldap.user_dn = uid=%u,ou=People,dc=example,dc=com

auth.ldap.ssl = false

Enable LDAP plugin:

./bin/emqttd_ctl plugins load emq_auth_ldap

HTTP

etc/plugins/emq_auth_http.conf:

## Variables: %u = username, %c = clientid, %a = ipaddress, %P = password, %t = topic

auth.http.auth_req = https://127.0.0.1:8080/mqtt/auth
auth.http.auth_req.method = post
auth.http.auth_req.params = clientid=%c,username=%u,password=%P

auth.http.super_req = https://127.0.0.1:8080/mqtt/superuser
auth.http.super_req.method = post
auth.http.super_req.params = clientid=%c,username=%u

Enable HTTP Plugin:

./bin/emqttd_ctl plugins load emq_auth_http

MySQL

Authenticate with MySQL database. Suppose that we create a mqtt_user table:

CREATE TABLE `mqtt_user` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `username` varchar(100) DEFAULT NULL,
  `password` varchar(100) DEFAULT NULL,
  `salt` varchar(20) DEFAULT NULL,
  `created` datetime DEFAULT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `mqtt_username` (`username`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

Configure the ‘auth_query’ and ‘password_hash’ in etc/plugins/emq_auth_mysql.conf:

## Mysql Server
auth.mysql.server = 127.0.0.1:3306

## Mysql Pool Size
auth.mysql.pool = 8

## Mysql Username
## auth.mysql.username =

## Mysql Password
## auth.mysql.password =

## Mysql Database
auth.mysql.database = mqtt

## Variables: %u = username, %c = clientid

## Authentication Query: select password only
auth.mysql.auth_query = select password from mqtt_user where username = '%u' limit 1

## Password hash: plain, md5, sha, sha256, pbkdf2
auth.mysql.password_hash = sha256

## %% Superuser Query
auth.mysql.super_query = select is_superuser from mqtt_user where username = '%u' limit 1

Enable MySQL plugin:

./bin/emqttd_ctl plugins load emq_auth_mysql

PostgreSQL

Authenticate with PostgreSQL database. Create a mqtt_user table:

CREATE TABLE mqtt_user (
  id SERIAL primary key,
  username character varying(100),
  password character varying(100),
  salt character varying(40)
);

Configure the ‘auth_query’ and ‘password_hash’ in etc/plugins/emq_auth_pgsql.conf:

## Postgre Server
auth.pgsql.server = 127.0.0.1:5432

auth.pgsql.pool = 8

auth.pgsql.username = root

#auth.pgsql.password =

auth.pgsql.database = mqtt

auth.pgsql.encoding = utf8

auth.pgsql.ssl = false

## Variables: %u = username, %c = clientid, %a = ipaddress

## Authentication Query: select password only
auth.pgsql.auth_query = select password from mqtt_user where username = '%u' limit 1

## Password hash: plain, md5, sha, sha256, pbkdf2
auth.pgsql.password_hash = sha256

## sha256 with salt prefix
## auth.pgsql.password_hash = salt sha256

## sha256 with salt suffix
## auth.pgsql.password_hash = sha256 salt

## Superuser Query
auth.pgsql.super_query = select is_superuser from mqtt_user where username = '%u' limit 1

Enable the plugin:

./bin/emqttd_ctl plugins load emq_auth_pgsql

Redis

Authenticate with Redis. MQTT users could be stored in redis HASH, the key is “mqtt_user:<Username>”.

Configure ‘auth_cmd’ and ‘password_hash’ in etc/plugins/emq_auth_redis.conf:

## Redis Server
auth.redis.server = 127.0.0.1:6379

## Redis Pool Size
auth.redis.pool = 8

## Redis Database
auth.redis.database = 0

## Redis Password
## auth.redis.password =

## Variables: %u = username, %c = clientid

## Authentication Query Command
auth.redis.auth_cmd = HGET mqtt_user:%u password

## Password hash: plain, md5, sha, sha256, pbkdf2
auth.redis.password_hash = sha256

## Superuser Query Command
auth.redis.super_cmd = HGET mqtt_user:%u is_superuser

Enable the plugin:

./bin/emqttd_ctl plugins load emq_auth_redis

MongoDB

Create a mqtt_user collection:

{
    username: "user",
    password: "password hash",
    is_superuser: boolean (true, false),
    created: "datetime"
}

Configure super_query, auth_query in etc/plugins/emq_auth_mongo.conf:

## Mongo Server
auth.mongo.server = 127.0.0.1:27017

## Mongo Pool Size
auth.mongo.pool = 8

## Mongo User
## auth.mongo.user =

## Mongo Password
## auth.mongo.password =

## Mongo Database
auth.mongo.database = mqtt

## auth_query
auth.mongo.auth_query.collection = mqtt_user

auth.mongo.auth_query.password_field = password

auth.mongo.auth_query.password_hash = sha256

auth.mongo.auth_query.selector = username=%u

## super_query
auth.mongo.super_query.collection = mqtt_user

auth.mongo.super_query.super_field = is_superuser

auth.mongo.super_query.selector = username=%u

Enable the plugin:

./bin/emqttd_ctl plugins load emq_auth_mongo

ACL

The ACL of EMQ broker is responsible for authorizing MQTT clients to publish/subscribe topics.

The ACL rules define:

Allow|Deny Who Publish|Subscribe Topics

Access Control Module of EMQ broker will match the rules one by one:

          ---------              ---------              ---------
Client -> | Rule1 | --nomatch--> | Rule2 | --nomatch--> | Rule3 | --> Default
          ---------              ---------              ---------
              |                      |                      |
            match                  match                  match
             \|/                    \|/                    \|/
        allow | deny           allow | deny           allow | deny

Internal

The default ACL of EMQ broker is implemented by an ‘internal’ module.

Enable the ‘internal’ ACL module in etc/emq.conf:

## ACL nomatch
mqtt.acl_nomatch = allow

## Default ACL File
mqtt.acl_file = etc/acl.conf

The ACL rules of ‘internal’ module are defined in ‘etc/acl.conf’ file:

%% Allow 'dashboard' to subscribe '$SYS/#'
{allow, {user, "dashboard"}, subscribe, ["$SYS/#"]}.

%% Allow clients from localhost to subscribe any topics
{allow, {ipaddr, "127.0.0.1"}, pubsub, ["$SYS/#", "#"]}.

%% Deny clients to subscribe '$SYS#' and '#'
{deny, all, subscribe, ["$SYS/#", {eq, "#"}]}.

%% Allow all by default
{allow, all}.

HTTP API

ACL by HTTP API: https://github.com/emqtt/emq_auth_http

Configure etc/plugins/emq_auth_http.conf and enable the plugin:

## 'access' parameter: sub = 1, pub = 2
auth.http.acl_req = https://127.0.0.1:8080/mqtt/acl
auth.http.acl_req.method = get
auth.http.acl_req.params = access=%A,username=%u,clientid=%c,ipaddr=%a,topic=%t

MySQL

ACL with MySQL database. The mqtt_acl table and default data:

CREATE TABLE `mqtt_acl` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `allow` int(1) DEFAULT NULL COMMENT '0: deny, 1: allow',
  `ipaddr` varchar(60) DEFAULT NULL COMMENT 'IpAddress',
  `username` varchar(100) DEFAULT NULL COMMENT 'Username',
  `clientid` varchar(100) DEFAULT NULL COMMENT 'ClientId',
  `access` int(2) NOT NULL COMMENT '1: subscribe, 2: publish, 3: pubsub',
  `topic` varchar(100) NOT NULL DEFAULT '' COMMENT 'Topic Filter',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

INSERT INTO mqtt_acl (id, allow, ipaddr, username, clientid, access, topic)
VALUES
    (1,1,NULL,'$all',NULL,2,'#'),
    (2,0,NULL,'$all',NULL,1,'$SYS/#'),
    (3,0,NULL,'$all',NULL,1,'eq #'),
    (5,1,'127.0.0.1',NULL,NULL,2,'$SYS/#'),
    (6,1,'127.0.0.1',NULL,NULL,2,'#'),
    (7,1,NULL,'dashboard',NULL,1,'$SYS/#');

Configure ‘acl-query’ and ‘acl_nomatch’ in etc/plugins/emq_auth_mysql.conf:

## ACL Query Command
auth.mysql.acl_query = select allow, ipaddr, username, clientid, access, topic from mqtt_acl where ipaddr = '%a' or username = '%u' or username = '$all' or clientid = '%c'

PostgreSQL

ACL with PostgreSQL database. The mqtt_acl table and default data:

CREATE TABLE mqtt_acl (
  id SERIAL primary key,
  allow integer,
  ipaddr character varying(60),
  username character varying(100),
  clientid character varying(100),
  access  integer,
  topic character varying(100)
);

INSERT INTO mqtt_acl (id, allow, ipaddr, username, clientid, access, topic)
VALUES
    (1,1,NULL,'$all',NULL,2,'#'),
    (2,0,NULL,'$all',NULL,1,'$SYS/#'),
    (3,0,NULL,'$all',NULL,1,'eq #'),
    (5,1,'127.0.0.1',NULL,NULL,2,'$SYS/#'),
    (6,1,'127.0.0.1',NULL,NULL,2,'#'),
    (7,1,NULL,'dashboard',NULL,1,'$SYS/#');

Configure ‘acl_query’ and ‘acl_nomatch’ in etc/plugins/emq_auth_pgsql.conf:

## ACL Query. Comment this query, the acl will be disabled.
auth.pgsql.acl_query = select allow, ipaddr, username, clientid, access, topic from mqtt_acl where ipaddr = '%a' or username = '%u' or username = '$all' or clientid = '%c'

Redis

ACL with Redis. The ACL rules are stored in a Redis HashSet:

HSET mqtt_acl:<username> topic1 1
HSET mqtt_acl:<username> topic2 2
HSET mqtt_acl:<username> topic3 3

Configure acl_cmd and acl_nomatch in etc/plugins/emq_auth_redis.conf:

## ACL Query Command
auth.redis.acl_cmd = HGETALL mqtt_acl:%u

MongoDB

Store ACL Rules in a mqtt_acl collection:

{
    username: "username",
    clientid: "clientid",
    publish: ["topic1", "topic2", ...],
    subscribe: ["subtop1", "subtop2", ...],
    pubsub: ["topic/#", "topic1", ...]
}

For example, insert rules into mqtt_acl collection:

db.mqtt_acl.insert({username: "test", publish: ["t/1", "t/2"], subscribe: ["user/%u", "client/%c"]})
db.mqtt_acl.insert({username: "admin", pubsub: ["#"]})

Configure acl_query and acl_nomatch in etc/plugins/emq_auth_mongo.conf:

## acl_query
auth.mongo.acl_query.collection = mqtt_user

auth.mongo.acl_query.selector = username=%u

MQTT Publish/Subscribe

MQTT is a an extremely lightweight publish/subscribe messaging protocol desgined for IoT, M2M and Mobile applications.

_images/pubsub_concept.png

Install and start the EMQ broker, and then any MQTT client could connect to the broker, subscribe topics and publish messages.

MQTT Client Libraries: https://github.com/mqtt/mqtt.github.io/wiki/libraries

For example, we use mosquitto_sub/pub commands:

mosquitto_sub -t topic -q 2
mosquitto_pub -t topic -q 1 -m "Hello, MQTT!"

MQTT V3.1.1 Protocol Specification: https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.html

MQTT Listener of the EMQ broker is configured in etc/emq.conf:

## TCP Listener: 1883, 127.0.0.1:1883, ::1:1883
listener.tcp.external = 1883

## Size of acceptor pool
listener.tcp.external.acceptors = 8

## Maximum number of concurrent clients
listener.tcp.external.max_clients = 1024

MQTT(SSL) Listener, Default Port is 8883:

## SSL Listener: 8883, 127.0.0.1:8883, ::1:8883
listener.ssl.external = 8883

## Size of acceptor pool
listener.ssl.external.acceptors = 4

## Maximum number of concurrent clients
listener.ssl.external.max_clients = 512

HTTP Publish API

The EMQ broker provides a HTTP API to help application servers publish messages to MQTT clients.

HTTP API: POST https://host:8080/mqtt/publish

Web servers such as PHP, Java, Python, NodeJS and Ruby on Rails could use HTTP POST to publish MQTT messages to the broker:

curl -v --basic -u user:passwd -d "qos=1&retain=0&topic=/a/b/c&message=hello from http..." -k http://localhost:8080/mqtt/publish

Parameters of the HTTP API:

Name Description
client clientid
qos QoS(0, 1, 2)
retain Retain(0, 1)
topic Topic
message Payload

Note

The API uses HTTP Basic Authentication.

The url of this API has been changed to ‘api/v2/mqtt/publish’ in v2.3-beta.2 release. Read the doc in REST API.

MQTT Over WebSocket

Web browsers could connect to the emqttd broker directly by MQTT Over WebSocket.

WebSocket URI: ws(s)://host:8083/mqtt
Sec-WebSocket-Protocol: ‘mqttv3.1’ or ‘mqttv3.1.1’

The Dashboard plugin provides a test page for WebSocket:

http://127.0.0.1:18083/websocket.html

Listener of WebSocket and HTTP Publish API is configured in etc/emq.config:

## MQTT/WebSocket Listener
listener.ws.external = 8083
listener.ws.external.acceptors = 4
listener.ws.external.max_clients = 64

$SYS Topics

The EMQ broker periodically publishes internal status, MQTT statistics, metrics and client online/offline status to $SYS/# topics.

For the EMQ broker could be clustered, the $SYS topic path is started with:

$SYS/brokers/${node}/

‘${node}’ is the erlang node name of emqttd broker. For example:

$SYS/brokers/emqttd@127.0.0.1/version

$SYS/brokers/emqttd@host2/uptime

Note

The broker only allows clients from localhost to subscribe $SYS topics by default.

Sys Interval of publishing $SYS messages, could be configured in etc/emqttd.config:

## System Interval of publishing broker $SYS Messages
mqtt.broker.sys_interval = 60

Broker Version, Uptime and Description

Topic Description
$SYS/brokers Broker nodes
$SYS/brokers/${node}/version Broker Version
$SYS/brokers/${node}/uptime Broker Uptime
$SYS/brokers/${node}/datetime Broker DateTime
$SYS/brokers/${node}/sysdescr Broker Description

Online/Offline Status of MQTT Client

The topic path started with: $SYS/brokers/${node}/clients/

Topic Payload(JSON) Description
${clientid}/connected
{ipaddress: “127.0.0.1”, username: “test”,
session: false, version: 3, connack: 0, ts: 1432648482}
Publish when a client connected
${clientid}/disconnected
{reason: “keepalive_timeout”,
username: “test”, ts: 1432749431}
Publish when a client disconnected

Properties of ‘connected’ Payload:

ipaddress: "127.0.0.1",
username:  "test",
session:   false,
protocol:  3,
connack:   0,
ts:        1432648482

Properties of ‘disconnected’ Payload:

reason: normal,
ts:     1432648486

Broker Statistics

Topic path started with: $SYS/brokers/${node}/stats/

Clients

Topic Description
clients/count Count of current connected clients
clients/max Max number of cocurrent connected clients

Sessions

Topic Description
sessions/count Count of current sessions
sessions/max Max number of sessions

Subscriptions

Topic Description
subscriptions/count Count of current subscriptions
subscriptions/max Max number of subscriptions

Topics

Topic Description
topics/count Count of current topics
topics/max Max number of topics

Broker Metrics

Topic path started with: $SYS/brokers/${node}/metrics/

Bytes Sent/Received

Topic Description
bytes/received MQTT Bytes Received since broker started
bytes/sent MQTT Bytes Sent since the broker started

Packets Sent/Received

Topic Description
packets/received MQTT Packets received
packets/sent MQTT Packets sent
packets/connect MQTT CONNECT Packet received
packets/connack MQTT CONNACK Packet sent
packets/publish/received MQTT PUBLISH packets received
packets/publish/sent MQTT PUBLISH packets sent
packets/subscribe MQTT SUBSCRIBE Packets received
packets/suback MQTT SUBACK packets sent
packets/unsubscribe MQTT UNSUBSCRIBE Packets received
packets/unsuback MQTT UNSUBACK Packets sent
packets/pingreq MQTT PINGREQ packets received
packets/pingresp MQTT PINGRESP Packets sent
packets/disconnect MQTT DISCONNECT Packets received

Messages Sent/Received

Topic Description
messages/received Messages Received
messages/sent Messages Sent
messages/retained Messages Retained
messages/stored TODO: Messages Stored
messages/dropped Messages Dropped

Broker Alarms

Topic path started with: $SYS/brokers/${node}/alarms/

Topic Description
${alarmId}/alert New Alarm
${alarmId}/clear Clear Alarm

Broker Sysmon

Topic path started with: ‘$SYS/brokers/${node}/sysmon/’

Topic Description
long_gc Long GC Warning
long_schedule Long Schedule
large_heap Large Heap Warning
busy_port Busy Port Warning
busy_dist_port Busy Dist Port

Trace

The emqttd broker supports to trace MQTT packets received/sent from/to a client, or trace MQTT messages published to a topic.

Trace a client:

./bin/emqttd_ctl trace client "clientid" "trace_clientid.log"

Trace a topic:

./bin/emqttd_ctl trace topic "topic" "trace_topic.log"

Lookup Traces:

./bin/emqttd_ctl trace list

Stop a Trace:

./bin/emqttd_ctl trace client "clientid" off

./bin/emqttd_ctl trace topic "topic" off