„README.md“ ändern
This commit is contained in:
Mario Voigt
parent
3b92306bcd
commit
15f8b7f1bd
616
README.md
616
README.md
@ -1,202 +1,486 @@
|
|||||||
# About
|
# Nukiana
|
||||||
Due to the lack of good description on how to setup Koel i decided to publish this manual because the installation of Koel required some hours for myself. I hope you will come to a result quicker than me with this cooking recipe ;-)
|
|
||||||
|
|
||||||
Some details of my platform:
|
* this page contains an overview of importing Nuki Web + Bridge into MariaDB by parsing JSON to csv
|
||||||
Operating system: Raspbian GNU/Linux 9.4 (stretch)
|
* the Nuki Web data comes directly from web.nuki.io to git-doid
|
||||||
Kernel: 4.14.69-v7+
|
* the Nuki Brdige data comes fromes pulling .json data from monitory-pi
|
||||||
Architecture: armv7l
|
|
||||||
Model: Raspberry Pi 3 Model B Rev 1.2
|
## 1. State of documentation
|
||||||
|
|
||||||
|
This project was born at 02.08.2018 and a first working prototype was made up for public at 09.08.2018. It is still really imperfect and not performant. But it works! I hope you will enjoy the work and maybe have ideas on how to improve the project.
|
||||||
|
|
||||||
|
All things were tested with ...
|
||||||
|
|
||||||
|
* Ubuntu 18.04 LTS for armhf v7 (Odroid X4)
|
||||||
|
* JQ 1.5
|
||||||
|
* Grafana 5.2.2
|
||||||
|
* MariaDB 10.1.34
|
||||||
|
* Nuki Bridge API 1.7 (30.07.2018)
|
||||||
|
* Nuki Web API 1.1 (08.05.2018)
|
||||||
|
* Nuki Bridge Firmware 1.11.0
|
||||||
|
* Nuki Bridge WiFi Firmware 1.2.0
|
||||||
|
* Nuki Smartlock Firmware 67074
|
||||||
|
* Nuki Swagger API → https://api.nuki.io (here you can find documentation on data models)
|
||||||
|
* Nuki Web → https://web.nuki.io (here you can retrieve your personal API key)
|
||||||
|
|
||||||
|
## 2. Requirements to utilize this project for your own needs
|
||||||
|
|
||||||
|
There are a few requirements you will need to fulfill to make this thing work. You need …
|
||||||
|
|
||||||
|
* a running Grafana instance with access to it
|
||||||
|
* i am using Grafana v5.2.2
|
||||||
|
* a Linux machine (virtual/bare metal) without super cow powers - this application is not cost intense
|
||||||
|
* i am using Ubuntu 18.04 LTS on an Odroid X4
|
||||||
|
* some basic knowledge about …
|
||||||
|
* accessing servers via SSH / working with console
|
||||||
|
* bash scripts
|
||||||
|
* administering MariaDB server (installation, starting, stopping, creating tables and users)
|
||||||
|
* basic knowledge about JSON format
|
||||||
|
* basic knowledge on how to work with Grafana Dashboards and Datasources
|
||||||
|
* network configuration: Port Forwarding, DynDNS, NAT traversal, SSH tunnels, iptables
|
||||||
|
|
||||||
|
## 3. Limitations
|
||||||
|
|
||||||
|
The current setup of this scripts is made for one Nuki Web Account which may include one or more Smartlock. In case that there are many Smartlocks under one Web Account it should grab information from all Locks automatically. If you want to monitor more than one Nuki Web Account you will need to setup the same script architecture more than once. The same is for the Nuki Bridge. Each bridge will require its own script set. BUT: You only need one database! You may use the same database for everything but if you like you also can add a datasource per Smartlock.
|
||||||
|
|
||||||
|
## 4. Prerequisites
|
||||||
|
### 4.1 JQ JSON parser
|
||||||
|
JQ is a really nice tool to parse JSON data. It also allows to export/convert data to other file formats. In this case we use the csv filter from JQ. By the way you can use https://jqplay.org/ to test different syntax without the need of a SSH shell
|
||||||
|
|
||||||
## Clone Repository
|
|
||||||
```
|
```
|
||||||
cd /opt
|
#Install JQ to convert nuki-web.json to nuki-web.csv
|
||||||
git clone https://github.com/phanan/koel.git
|
apt-get install jq
|
||||||
cd koel
|
|
||||||
git checkout v3.7.2
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Install Software Basics
|
### 4.2 Database nuki
|
||||||
```
|
```
|
||||||
apt-get install php php-xml php-mbstring php-curl php-zip php-pgsql postgresql apache2
|
#Install MariaDB + Configure it
|
||||||
|
apt-get install mariadb-server
|
||||||
```
|
```
|
||||||
|
|
||||||
## Install NodeJS
|
|
||||||
```
|
```
|
||||||
curl -sL https://deb.nodesource.com/setup_10.x | bash -
|
/*this table is a merge of /smartlock/; /smartlock/config/ and /smartlock/advanced/config/*/
|
||||||
apt-get install -y nodejs
|
CREATE TABLE IF NOT EXISTS Smartlock
|
||||||
```
|
(
|
||||||
|
`accountId` VARCHAR(9) NOT NULL,
|
||||||
## Install Yarn
|
`adminPinState` TINYINT NOT NULL,
|
||||||
```
|
`advertisingMode` TINYINT NOT NULL,
|
||||||
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
|
`authId` VARCHAR(24),
|
||||||
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
|
`autoLockTimeout` SMALLINT NOT NULL,
|
||||||
apt-get update
|
`autoUnlatch` BOOLEAN NOT NULL,
|
||||||
apt-get install yarn
|
`automaticBatteryTypeDetection` BOOLEAN NOT NULL,
|
||||||
```
|
`batteryCritical` BOOLEAN NOT NULL,
|
||||||
|
`batteryType` TINYINT NOT NULL,
|
||||||
## Get Composer
|
`buttonEnabled` BOOLEAN NOT NULL,
|
||||||
```
|
`creationDate` DATETIME NOT NULL,
|
||||||
php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
|
`daylightSavingMode` TINYINT NOT NULL,
|
||||||
php -r "if (hash_file('SHA384', 'composer-setup.php') === '544e09ee996cdf60ece3804abc52599c22b1f40f4323403c44d44fdfdd586475ca9813a858088ffbc1f233e9b180f061') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;"
|
`detachedCylinder` BOOLEAN NOT NULL,
|
||||||
php composer-setup.php
|
`doubleButtonPressAction` TINYINT NOT NULL,
|
||||||
php -r "unlink('composer-setup.php');"
|
`favorite` BOOLEAN NOT NULL,
|
||||||
```
|
`firmwareVersion` VARCHAR(255),
|
||||||
|
`fobAction1` TINYINT NOT NULL,
|
||||||
## Use Composer
|
`fobAction2` TINYINT NOT NULL,
|
||||||
```
|
`fobAction3` TINYINT NOT NULL,
|
||||||
php composer.phar install
|
`fobPaired` BOOLEAN NOT NULL,
|
||||||
```
|
`keypadPaired` BOOLEAN NOT NULL,
|
||||||
|
`lastAction` TINYINT NOT NULL,
|
||||||
## Make Postgres Database
|
`latitude` FLOAT NOT NULL,
|
||||||
```
|
`ledBrightness` TINYINT NOT NULL,
|
||||||
sudo - postgres
|
`ledEnabled` BOOLEAN NOT NULL,
|
||||||
createuser --no-superuser --pwprompt koel
|
`lngTimeout` SMALLINT NOT NULL,
|
||||||
psql
|
`lockedPositionOffsetDegrees` SMALLINT NOT NULL,
|
||||||
CREATE database koel WITH OWNER koel ENCODING 'utf8' LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8';
|
`longitude` FLOAT NOT NULL,
|
||||||
GRANT ALL PRIVILEGES ON DATABASE koel TO koel;
|
`mainName` VARCHAR(255),
|
||||||
|
`mode` TINYINT NOT NULL,
|
||||||
|
`name` VARCHAR(255),
|
||||||
|
`pairingEnabled` BOOLEAN NOT NULL,
|
||||||
|
`serverState` TINYINT NOT NULL,
|
||||||
|
`singleButtonPressAction` TINYINT NOT NULL,
|
||||||
|
`singleLock` BOOLEAN NOT NULL,
|
||||||
|
`singleLockedPositionOffsetDegrees` SMALLINT NOT NULL,
|
||||||
|
`smartlockId` VARCHAR(9) NOT NULL PRIMARY KEY, /*primary key = unique!*/
|
||||||
|
`state` TINYINT NOT NULL,
|
||||||
|
`timezoneOffset` SMALLINT NOT NULL,
|
||||||
|
`totalDegrees` SMALLINT NOT NULL,
|
||||||
|
`trigger` TINYINT NOT NULL,
|
||||||
|
`type` TINYINT NOT NULL,
|
||||||
|
`unlatchDuration` SMALLINT NOT NULL,
|
||||||
|
`unlockedPositionOffsetDegrees` SMALLINT NOT NULL,
|
||||||
|
`unlockedToLockedTransitionOffsetDegrees` SMALLINT NOT NULL,
|
||||||
|
`updateDate` DATETIME NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS SmartlockLog
|
||||||
|
(
|
||||||
|
`action` TINYINT NOT NULL,
|
||||||
|
`authId` VARCHAR(24),
|
||||||
|
`autoUnlock` BOOLEAN NOT NULL,
|
||||||
|
`date` DATETIME NOT NULL,
|
||||||
|
`id` VARCHAR(24) NOT NULL PRIMARY KEY, /*primary key = unique!*/
|
||||||
|
`name` VARCHAR(255),
|
||||||
|
`smartlockId` VARCHAR(9) NOT NULL,
|
||||||
|
`state` TINYINT NOT NULL,
|
||||||
|
`trigger` TINYINT NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS SmartlockAuth
|
||||||
|
(
|
||||||
|
`authId` VARCHAR(24), /*it may be shorter -> 3 digits*/
|
||||||
|
`creationDate` DATETIME NOT NULL,
|
||||||
|
`enabled` BOOLEAN NOT NULL,
|
||||||
|
`id` VARCHAR(24) NOT NULL PRIMARY KEY, /*primary key = unique!*/
|
||||||
|
`lastActiveDate` DATETIME NOT NULL,
|
||||||
|
`lockCount` SMALLINT NOT NULL,
|
||||||
|
`name` VARCHAR(255),
|
||||||
|
`remoteAllowed` BOOLEAN NOT NULL,
|
||||||
|
`smartlockId` VARCHAR(9) NOT NULL,
|
||||||
|
`type` TINYINT NOT NULL,
|
||||||
|
`updateDate` DATETIME NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
|
||||||
|
/*look script: you need to insert the nukiId in jq*/
|
||||||
|
CREATE TABLE IF NOT EXISTS BridgeCallbackList
|
||||||
|
(
|
||||||
|
`id` TINYINT NOT NULL,
|
||||||
|
`nukiId` VARCHAR(9) NOT NULL,
|
||||||
|
`url` TEXT NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS BridgeInfo
|
||||||
|
(
|
||||||
|
`appVersion` VARCHAR(24),
|
||||||
|
`bridgeType` TINYINT NOT NULL,
|
||||||
|
`currentTime` DATETIME NOT NULL,
|
||||||
|
`firmwareVersion` VARCHAR(24),
|
||||||
|
`hardwareId` VARCHAR(24),
|
||||||
|
`name` VARCHAR(255) NOT NULL,
|
||||||
|
`nukiId` VARCHAR(9) NOT NULL PRIMARY KEY, /*primary key = unique!*/
|
||||||
|
`paired` TINYINT NOT NULL,
|
||||||
|
`rssi` TINYINT NOT NULL,
|
||||||
|
`serverConnected` TINYINT NOT NULL,
|
||||||
|
`serverId` VARCHAR(24) NOT NULL,
|
||||||
|
`uptime` BIGINT NOT NULL,
|
||||||
|
`wifiFirmwareVersion` VARCHAR(24)
|
||||||
|
);
|
||||||
|
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS BridgeList
|
||||||
|
(
|
||||||
|
`batteryCritical` BOOLEAN NOT NULL,
|
||||||
|
`name` TINYINT NOT NULL,
|
||||||
|
`nukiId` VARCHAR(9) NOT NULL PRIMARY KEY, /*primary key = unique!*/
|
||||||
|
`state` TINYINT NOT NULL,
|
||||||
|
`stateName` VARCHAR(24) NOT NULL,
|
||||||
|
`timestamp` DATETIME NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
|
||||||
|
/*look script: you need to insert the nukiId in jq*/
|
||||||
|
CREATE TABLE IF NOT EXISTS BridgeLockState
|
||||||
|
(
|
||||||
|
`batteryCritical` BOOLEAN NOT NULL,
|
||||||
|
`nukiId` VARCHAR(9) NOT NULL PRIMARY KEY, /*primary key = unique!*/
|
||||||
|
`state` TINYINT NOT NULL,
|
||||||
|
`stateName` VARCHAR(24) NOT NULL,
|
||||||
|
`success` BOOLEAN NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
|
||||||
|
/*look script: you need to insert the bridgeId (=nukiId) in jq*/
|
||||||
|
CREATE TABLE IF NOT EXISTS BridgeLog
|
||||||
|
(
|
||||||
|
`bridgeId` VARCHAR(9) NOT NULL,
|
||||||
|
`cmdId` VARCHAR(9),
|
||||||
|
`connection` TINYINT,
|
||||||
|
`macAddr` VARCHAR(12),
|
||||||
|
`nukiId` VARCHAR(9),
|
||||||
|
`timestamp` DATETIME NOT NULL,
|
||||||
|
`type` VARCHAR(255) NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
\q
|
\q
|
||||||
```
|
```
|
||||||
|
|
||||||
## Edit php config
|
## 5. Nuki parsing + conversion
|
||||||
|
### 5.1 Smartlock Data
|
||||||
|
|
||||||
```
|
```
|
||||||
vim /etc/php/7.0/apache2/php.ini
|
mkdir -p /opt/nuki/web
|
||||||
#enable pdo_pgsql and pgsql extensions - it's questionable if this is really required
|
chmod -R o-rwx /opt/nuki/ #disable access for other users except user/group
|
||||||
extension=php_pdo_pgsql.dll
|
chmod -R ug+rwx /opt/nuki/
|
||||||
extension=php_pfsql.dll
|
|
||||||
memory_limit=512M
|
vim /opt/nuki/web/nuki_web_job.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
## Run npm/yarn
|
> /opt/nuki/web/nuki_web_job.sh
|
||||||
```
|
```
|
||||||
vim /opt/koel/packages.json #replace ^4.7.2 with ~4.7.2 for node-sass
|
#!/bin/bash
|
||||||
|
|
||||||
npm update
|
NUKI_ROOT="/opt/nuki/web"
|
||||||
npm install --unsafe-perm=true --allow-root
|
cd $NUKI_ROOT
|
||||||
```
|
|
||||||
|
|
||||||
## Setup Koel
|
|
||||||
```
|
|
||||||
# Populate credentials during the process
|
|
||||||
php artisan koel:init
|
|
||||||
|
|
||||||
Koel cannot connect to the database. Let's set it up.
|
API_TOKEN="YourPersonal80CharacterApiToken"
|
||||||
|
|
||||||
Your DB driver of choice [MySQL/MariaDB]:
|
CURL_HEADER="--header 'Accept: application/json' --header 'Authorization: Bearer $API_TOKEN'"
|
||||||
[mysql ] MySQL/MariaDB
|
BASE_URL="https://api.nuki.io"
|
||||||
[pgsql ] PostgreSQL
|
JQ_CSV='(map(keys)|add|unique) as $cols|map(. as $row|$cols|map($row[.])) as $rows|$cols, $rows[]|@csv'
|
||||||
[sqlsrv ] SQL Server
|
FILE_BASE="nuki_web"
|
||||||
[sqlite-e2e] SQLite
|
|
||||||
> pgsql
|
|
||||||
|
|
||||||
DB host:
|
DB_PASS="YourPassword"
|
||||||
> localhost
|
DB_USER="nuki"
|
||||||
|
DB_NAME="nuki"
|
||||||
|
|
||||||
DB port (leave empty for default) []:
|
while [ true ]; do
|
||||||
> 5432
|
|
||||||
|
|
||||||
DB name:
|
eval "curl -X GET $CURL_HEADER $BASE_URL/smartlock/log -o "$FILE_BASE"_SmartlockLog.json"
|
||||||
> koel
|
jq -r "$JQ_CSV" "$FILE_BASE"_SmartlockLog.json > "$FILE_BASE"_SmartlockLog.csv
|
||||||
|
sed -i 's/,true,/,1,/g' "$FILE_BASE"_SmartlockLog.csv
|
||||||
|
sed -i 's/,false,/,0,/g' "$FILE_BASE"_SmartlockLog.csv
|
||||||
|
mysql -u$DB_USER -p$DB_PASS $DB_NAME -e"TRUNCATE TABLE SmartlockLog; LOAD DATA LOCAL INFILE '"$FILE_BASE"_SmartlockLog.csv' INTO TABLE SmartlockLog FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\n' IGNORE 1 LINES;"
|
||||||
|
|
||||||
DB user:
|
eval "curl -X GET $CURL_HEADER $BASE_URL/smartlock/auth -o "$FILE_BASE"_SmartlockAuth.json"
|
||||||
> koel
|
jq -r "$JQ_CSV" "$FILE_BASE"_SmartlockAuth.json > "$FILE_BASE"_SmartlockAuth.csv
|
||||||
|
sed -i 's/,true,/,1,/g' "$FILE_BASE"_SmartlockAuth.csv
|
||||||
|
sed -i 's/,false,/,0,/g' "$FILE_BASE"_SmartlockAuth.csv
|
||||||
|
mysql -u$DB_USER -p$DB_PASS $DB_NAME -e"TRUNCATE TABLE SmartlockAuth; LOAD DATA LOCAL INFILE '"$FILE_BASE"_SmartlockAuth.csv' INTO TABLE SmartlockAuth FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\n' IGNORE 1 LINES;"
|
||||||
|
|
||||||
DB password []:
|
eval "curl -X GET $CURL_HEADER $BASE_URL/smartlock -o "$FILE_BASE"_Smartlock.json"
|
||||||
> nZ85pEkeQPhJ1458YGre
|
jq -r ".[]|[{smartlockId}+{accountId}+{type}+{authId}+({\"mainName\":.name}|del(.name))+{favorite}+{firmwareVersion}+{serverState}+{adminPinState}+{creationDate}+{updateDate}+.config+.advancedConfig+.state]|$JQ_CSV" "$FILE_BASE"_Smartlock.json > "$FILE_BASE"_Smartlock.csv
|
||||||
|
sed -i 's/true/1/g' "$FILE_BASE"_Smartlock.csv
|
||||||
|
sed -i 's/false/0/g' "$FILE_BASE"_Smartlock.csv
|
||||||
|
mysql -u$DB_USER -p$DB_PASS $DB_NAME -e"TRUNCATE TABLE Smartlock; LOAD DATA LOCAL INFILE '"$FILE_BASE"_Smartlock.csv' INTO TABLE Smartlock FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\n' IGNORE 1 LINES;"
|
||||||
|
|
||||||
Migrating database
|
sleep 30
|
||||||
Let's create the admin account.
|
done
|
||||||
|
|
||||||
Your name:
|
|
||||||
> Administrator
|
|
||||||
|
|
||||||
Your email address:
|
|
||||||
> admin@admin.de
|
|
||||||
|
|
||||||
Your desired password:
|
|
||||||
>
|
|
||||||
|
|
||||||
Again, just to make sure:
|
|
||||||
>
|
|
||||||
|
|
||||||
Seeding initial data
|
|
||||||
The absolute path to your media directory. If this is skipped (left blank) now, you can set it later via the web interface.
|
|
||||||
|
|
||||||
Media path []:
|
|
||||||
> /mnt/shared_media
|
|
||||||
|
|
||||||
Compiling front-end stuff
|
|
||||||
sh: 1: yarn: not found
|
|
||||||
|
|
||||||
🎆 Success! Koel can now be run from localhost with `php artisan serve`.
|
|
||||||
You can also scan for media with `php artisan koel:sync`.
|
|
||||||
Again, for more configuration guidance, refer to
|
|
||||||
📙 https://koel.phanan.net/docs
|
|
||||||
or open the .env file in the root installation folder.
|
|
||||||
Thanks for using Koel. You rock!
|
|
||||||
```
|
|
||||||
|
|
||||||
## Apache Reverse Proxy
|
|
||||||
```
|
|
||||||
<VirtualHost *:80>
|
|
||||||
RewriteEngine on
|
|
||||||
RewriteCond %{HTTPS} !^on$ [NC]
|
|
||||||
RewriteRule . https://%{HTTP_HOST}%{REQUEST_URI} [L]
|
|
||||||
</VirtualHost>
|
|
||||||
<VirtualHost *:443>
|
|
||||||
ServerName music.yourdomain.de
|
|
||||||
Header set Access-Control-Allow-Origin "https://music.yourdomain.de"
|
|
||||||
Header set Access-Control-Allow-Headers "x-requested-with, Content-Type, origin, authorization, accept, client-security-token"
|
|
||||||
SSLEngine on
|
|
||||||
SSLVerifyClient none
|
|
||||||
SSLCertificateFile /etc/ssl/yourdomain.de.pem
|
|
||||||
|
|
||||||
ProxyPass / http://127.0.0.1:8000/
|
|
||||||
ProxyPassReverse / http://127.0.0.1:8000/
|
|
||||||
</VirtualHost>
|
|
||||||
```
|
```
|
||||||
|
|
||||||
```
|
```
|
||||||
a2enmod alias rewrite proxy proxy_html proxy_http ssl vhost_alias xml2enc
|
chmod 770 nuki_web_job.sh
|
||||||
a2enmod music.yourdomain.de.conf
|
|
||||||
apachectl configtest
|
#test the script
|
||||||
service apache2 restart
|
./nuki_web_job.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
## Dirty hack to enable https
|
### 5.2 Nuki Bridge Collector
|
||||||
|
Because my Database (Grafana Datasource) is on another network than the Nuki Bridge i needed to split up the installation. That means that i grab the json file from Nuki Bridge from a Raspberry Pi which is on the same network. By using some SSH user with public key authentication i am able to pull the data via Odroid.
|
||||||
|
|
||||||
|
#### 5.2.1 On the collector node side
|
||||||
|
The following scripts are placed on a server which is on the same network as the Nuki Bridge. You might open the port 8080 as well via firewall but i did not want to do this because the basic Nuki token is really short and unsafe (only 6 chars). The script will collect data from bridge and save it to /opt/nuki/bridge. A rsync script will grab this data.
|
||||||
|
|
||||||
```
|
```
|
||||||
vim /opt/koel/routes/web.php
|
mkdir -p /opt/nuki/bridge
|
||||||
|
chmod -R o-rwx /opt/nuki/ #disable access for other users except user/group
|
||||||
#add the URL:: lines to make it work
|
|
||||||
|
vim /opt/nuki/bridge/nuki_bridge_job-collector.sh
|
||||||
<?php
|
|
||||||
|
|
||||||
URL::forceRootUrl('https://sub.yourdomain.de:5443');
|
|
||||||
URL::forceScheme('https');
|
|
||||||
|
|
||||||
Route::get('/', function () {
|
|
||||||
return view('index');
|
|
||||||
});
|
|
||||||
|
|
||||||
// Some backward compatibilities.
|
|
||||||
Route::get('/♫', function () {
|
|
||||||
return redirect('/');
|
|
||||||
});
|
|
||||||
|
|
||||||
Route::get('/remote', function () {
|
|
||||||
return view('remote');
|
|
||||||
});
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Start Server
|
> /opt/nuki/bridge/nuki_bridge_job-collector.sh
|
||||||
```
|
```
|
||||||
#php artisan serve
|
#!/bin/bash
|
||||||
cd /opt/koel/;nohup php artisan serve --host 0.0.0.0 > koel-artisan-server.log &
|
|
||||||
|
NUKI_ROOT="/opt/nuki/bridge"
|
||||||
|
cd $NUKI_ROOT
|
||||||
|
|
||||||
|
CURL_HEADER="--header 'Accept: application/json'"
|
||||||
|
BASE_URL="http://<YourLocalIPAddress>:8080"
|
||||||
|
FILE_BASE="nuki_bridge"
|
||||||
|
TOKEN="Your6CharacterToken"
|
||||||
|
LOG_COUNT=100
|
||||||
|
LOG_OFFSET=0
|
||||||
|
NUKIID="Your8CharacterNukiID"
|
||||||
|
|
||||||
|
while [ true ]; do
|
||||||
|
eval "curl -X GET $CURL_HEADER '$BASE_URL/info?token=$TOKEN' -o "$FILE_BASE"_info.json"
|
||||||
|
eval "curl -X GET $CURL_HEADER '$BASE_URL/list?token=$TOKEN' -o "$FILE_BASE"_list.json"
|
||||||
|
eval "curl -X GET $CURL_HEADER '$BASE_URL/lockState?token=$TOKEN&nukIid=$NUKIID' -o "$FILE_BASE"_lockState.json"
|
||||||
|
eval "curl -X GET $CURL_HEADER '$BASE_URL/callback/list?token=$TOKEN' -o "$FILE_BASE"_callback-list.json"
|
||||||
|
eval "curl -X GET $CURL_HEADER '$BASE_URL/log?token=$TOKEN&count=$LOG_COUNT&offset=$LOG_OFFSET' -o "$FILE_BASE"_log.json"
|
||||||
|
sleep 30
|
||||||
|
done
|
||||||
```
|
```
|
||||||
|
|
||||||
## Version overview
|
|
||||||
```
|
```
|
||||||
php -V #7.0
|
chmod 770 nuki_bridge_job-collector.sh
|
||||||
npm --version #6.4.1
|
vim /etc/rc.local
|
||||||
nodejs v10.11.0
|
|
||||||
yarn --version #1.9.4
|
```
|
||||||
```
|
|
||||||
|
> /etc/rc.local
|
||||||
|
```
|
||||||
|
if [ -f /aafirstboot ]; then /aafirstboot start ; fi
|
||||||
|
|
||||||
|
if [ -f /aafirstboot ]; then /aafirstboot start ; fi
|
||||||
|
|
||||||
|
nohup /opt/nuki/bridge/nuki_bridge_job-collector.sh > /opt/nuki/bridge/nuki_bridge_job-collector.log 2>&1 &
|
||||||
|
|
||||||
|
exit 0
|
||||||
|
```
|
||||||
|
|
||||||
|
Init the modified rc.local and check status
|
||||||
|
```
|
||||||
|
sudu su
|
||||||
|
. /etc/rc.local
|
||||||
|
ps -afe | grep [n]uki
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 5.2.2 SSH sync user
|
||||||
|
```
|
||||||
|
#[collector node]
|
||||||
|
addgroup --gid 1111 sshbackup
|
||||||
|
adduser --gecos "" --uid 1111 --gid 1111 --disabled-password sshbackup
|
||||||
|
passwd sshbackup
|
||||||
|
|
||||||
|
#from any client - check if it works + copy pub key
|
||||||
|
ssh -o PubkeyAuthentication=no sshbackup@sshbackup@sub.doma.in -p 22 #with password
|
||||||
|
ssh-copy-id -o PubkeyAuthentication=no -f -i ~/.ssh/sshbackup sshbackup@sub.doma.in -p 22
|
||||||
|
|
||||||
|
#if something files with too many bad authentications check fail2ban
|
||||||
|
fail2ban-client status sshd
|
||||||
|
sudo grep 'Ban ' /var/log/fail2ban.log*
|
||||||
|
fail2ban-client set sshd unbanip <TheIP>
|
||||||
|
|
||||||
|
#[collector node]
|
||||||
|
chsh -s /bin/bash sshbackup
|
||||||
|
passwd -d sshbackup
|
||||||
|
|
||||||
|
#[parser node]
|
||||||
|
#copy sshtunnel private key to /home/pi/.ssh/sshtunnel
|
||||||
|
chmod 600 /home/pi/.ssh/sshbackup
|
||||||
|
|
||||||
|
#check if it works with public key
|
||||||
|
ssh -i ~/.ssh/sshbackup sshbackup@sub.doma.in -p 22
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 5.2.3 On the parsing node side
|
||||||
|
The following scripts are installed on the same server where MariaDB is installed.
|
||||||
|
```
|
||||||
|
mkdir -p /opt/nuki/bridge
|
||||||
|
chmod -R o-rwx /opt/nuki/ #disable access for other users except user/group
|
||||||
|
chmod -R ug+rwx /opt/nuki/
|
||||||
|
|
||||||
|
vim /opt/nuki/bridge/nuki_bridge_job-parser.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
> /opt/nuki/bridge/nuki_bridge_job-parser.sh
|
||||||
|
```
|
||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
NUKI_ROOT="/opt/nuki/bridge"
|
||||||
|
cd $NUKI_ROOT
|
||||||
|
|
||||||
|
JQ_CSV='(map(keys)|add|unique) as $cols|map(. as $row|$cols|map($row[.])) as $rows|$cols, $rows[]|@csv'
|
||||||
|
FILE_BASE="nuki_bridge"
|
||||||
|
NUKIID="Your8CharacterNukiID"
|
||||||
|
DB_PASS="YourPassword"
|
||||||
|
DB_USER="nuki"
|
||||||
|
DB_NAME="nuki"
|
||||||
|
|
||||||
|
REMOTE_HOST="sub.doma.in"
|
||||||
|
REMOTE_PORT="22"
|
||||||
|
REMOTE_USER="sshbackup"
|
||||||
|
REMOTE_PRIVKEY="~/.ssh/sshbackup"
|
||||||
|
|
||||||
|
while [ true ]; do
|
||||||
|
/usr/bin/rsync -avz -e "ssh -i $REMOTE_PRIVKEY -p $REMOTE_PORT" $REMOTE_USER@$REMOTE_HOST:$NUKI_ROOT/*.json $NUKI_ROOT/
|
||||||
|
|
||||||
|
|
||||||
|
jq -r "[{bridgeType}+(.|{ids}|{hardwareId}+.[])+(.|{versions}|{appVersion}+{firmwareVersion}+{wifiFirmwareVersion}+.[])+{uptime}+{currentTime}+{serverConnected}+(.|{scanResults}|.[][])]|$JQ_CSV" "$FILE_BASE"_info.json > "$FILE_BASE"_info.csv
|
||||||
|
sed -i 's/,true,/,1,/g' "$FILE_BASE"_info.csv
|
||||||
|
sed -i 's/,false,/,0,/g' "$FILE_BASE"_info.csv
|
||||||
|
mysql -u$DB_USER -p$DB_PASS $DB_NAME -e"TRUNCATE Table BridgeInfo; LOAD DATA LOCAL INFILE '"$FILE_BASE"_info.csv' INTO TABLE BridgeInfo FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\n' IGNORE 1 LINES;"
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
jq -r "[.[]|{nukiId}+{name}+({lastKnownState}|.[])]|$JQ_CSV" "$FILE_BASE"_list.json > "$FILE_BASE"_list.csv
|
||||||
|
sed -i 's/,true,/,1,/g' "$FILE_BASE"_list.csv
|
||||||
|
sed -i 's/,false,/,0,/g' "$FILE_BASE"_list.csv
|
||||||
|
mysql -u$DB_USER -p$DB_PASS $DB_NAME -e"TRUNCATE TABLE BridgeList; LOAD DATA LOCAL INFILE '"$FILE_BASE"_list.csv' INTO TABLE BridgeList FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\n' IGNORE 1 LINES;"
|
||||||
|
|
||||||
|
|
||||||
|
jq -r "[.+{nukiId:\"$NUKIID\"}]|$JQ_CSV" "$FILE_BASE"_lockState.json > "$FILE_BASE"_lockState.csv
|
||||||
|
sed -i 's/true/1/g' "$FILE_BASE"_lockState.csv
|
||||||
|
sed -i 's/false/0/g' "$FILE_BASE"_lockState.csv
|
||||||
|
mysql -u$DB_USER -p$DB_PASS $DB_NAME -e"TRUNCATE TABLE BridgeLockState; LOAD DATA LOCAL INFILE '"$FILE_BASE"_lockState.csv' INTO TABLE BridgeLockState FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\n' IGNORE 1 LINES;"
|
||||||
|
|
||||||
|
#will return "Cannot iterate over null (null)" if file is empty!
|
||||||
|
jq -r ".|{callbacks}|\"$NUKIID\" as \$a|.callbacks[].nukiId=\$a|.[]|$JQ_CSV" "$FILE_BASE"_callback-list.json > "$FILE_BASE"_callback-list.csv
|
||||||
|
mysql -u$DB_USER -p$DB_PASS $DB_NAME -e"TRUNCATE TABLE BridgeCallbackList; LOAD DATA LOCAL INFILE '"$FILE_BASE"_callback-list.csv' INTO TABLE BridgeCallbackList FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\n' IGNORE 1 LINES;"
|
||||||
|
|
||||||
|
|
||||||
|
jq -r "[.[]|{timestamp}+{type}+{connection}+{nukiId}+{cmdId}+{macAddr}+{bridgeId:\"$NUKIID\"}]|$JQ_CSV" "$FILE_BASE"_log.json > "$FILE_BASE"_log.csv
|
||||||
|
mysql -u$DB_USER -p$DB_PASS $DB_NAME -e"TRUNCATE TABLE BridgeLog; LOAD DATA LOCAL INFILE '"$FILE_BASE"_log.csv' INTO TABLE BridgeLog FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\n' IGNORE 1 LINES;"
|
||||||
|
|
||||||
|
|
||||||
|
sleep 30
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
```
|
||||||
|
chmod 770 nuki_bridge_job-parser.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
```
|
||||||
|
#test the script
|
||||||
|
./nuki_bridge_job-parser.sh
|
||||||
|
```
|
||||||
|
### 5.3 Putting the scripts to autostart (rc.local)
|
||||||
|
```
|
||||||
|
vim /etc/rc.local
|
||||||
|
```
|
||||||
|
|
||||||
|
> /etc/rc.local
|
||||||
|
```
|
||||||
|
if [ -f /aafirstboot ]; then /aafirstboot start ; fi
|
||||||
|
|
||||||
|
if [ -f /aafirstboot ]; then /aafirstboot start ; fi
|
||||||
|
|
||||||
|
nohup /opt/nuki/web/nuki_web_job.sh > /opt/nuki/web/nuki_web_job.log 2>&1 &
|
||||||
|
nohup /opt/nuki/bridge/nuki_bridge_job-parser.sh > /opt/nuki/bridge/nuki_bridge_job-parser.log 2>&1 &
|
||||||
|
|
||||||
|
exit 0
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.4 Test it!
|
||||||
|
```
|
||||||
|
#reload
|
||||||
|
. /etc/rc.local & bash -c "ps -afe | grep [n]uki"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.5 Customization parameters
|
||||||
|
|
||||||
|
* by adjusting the bash scripts you can modify the update frequency for CURLing and parsing json files (currently 30s is standard value)
|
||||||
|
* Bridge Id has to be added so the conjunction between Smartlock and Bridge can be established. This could be automated but at the moment it has to be done manually (replace in JQ parsing lines)
|
||||||
|
* you can customize offset and count for BridgeLog by redefining argument
|
||||||
|
|
||||||
|
## 6. Download Grafana Dashboard
|
||||||
|
https://grafana.com/dashboards/7628/ or https://leyghis.fablabchemnitz.de:8444/MarioVoigt/Nukiana/src/branch/master/Grafana/Nuki%20Smartlock%20+%20Bridge%20%28Internals%29.json
|
||||||
|
|
||||||
|
## 7. Remaining ToDo's
|
||||||
|
* bash scripts
|
||||||
|
* maybe add pid files / design them as services for "sudo service ... stop|start|restart"
|
||||||
|
* translation
|
||||||
|
* at the moment the code is in english and the dashboard is only available in german
|
||||||
|
* possible solutions:
|
||||||
|
* add a language drop down variable to Grafana + rewrite every select so strings get automatically translated → decimal separators should be rewritten too
|
||||||
|
* add extra translation columns to MariaDB tables + some kind of translation string repository to fill the columns → could be utilized by DB triggers/routines
|
||||||
|
* rewrite curl commands so that the log files (SmartlockLog, BridgeLog) are getting streamed until the point (Id) where the last entry was loaded previously to ...
|
||||||
|
* reduce bandwith
|
||||||
|
* speed up the process
|
||||||
|
* make the "sed" script lines nicer → i guess they can be completely removed by using JQ
|
||||||
|
* BridgeLog: "handles" are not imported and are getting ignored
|
||||||
|
* Bridge provides timestamp but no timezoneOffset → SQL statements were written to use the timezoneOffset of Smartlock instead → works, but dirty
|
||||||
|
* replace TRUNCATE statements with "INSERT ON DUPLICATE KEY UPDATE" to leave old data but import new data
|
||||||
|
* to implement:
|
||||||
|
* make us of fields `keypadPaired` and `lastAction`
|
||||||
|
* build up some check script to make proof of correct count of columns and column order of the csv files
|
||||||
|
* collect data with time history:
|
||||||
|
* history of firmware changes
|
||||||
|
* history of rssi values
|
||||||
|
* history of state changes: locking states, pairing state
|
||||||
|
* to test: this project was never tested with more than one Smartlock and more than one Bridge. We only have one! I also did not test what data output FOB will produce
|
||||||
|
* Maybe give a try to send the JSON output files directly to JQ/Filebeat → Elasticsearch → Grafana Datasource for Elasticsearch ↔ possible?
|
||||||
|
* Write some data collector tables like "BridgeFirmwareHistory" and "SmartlockFirmwareHistory" for tracking firmware changes utilizing some trigger
|
||||||
|
* BridgeFirmwareHistory: BridgeId, date, BridgeFirmware, BridgeWifiFirmware,BridgeAppVersion
|
||||||
|
* SmartlockFirmwareHistory: SmartlockId,date,SmartlockFirmware
|
||||||
Loading…
Reference in New Issue
Block a user