jaNET Framework... after all these years

It has been a while.

Development of Jubito and the jaNET Framework was discontinued back in 2017. The project remained online, the source code remained available and, from time to time, I could see that there were still people visiting the website, reading this blog, cloning the repository or simply experimenting with something that started many years ago.

Technology, of course, didn't stop there. .NET Framework and Mono evolved into modern .NET, Raspberry Pi became much more powerful, ESP32 became almost a standard component of DIY automation, MQTT is everywhere and many of the things we were experimenting with back then are today considered normal parts of an IoT environment.

So, recently I decided to do something I wasn't really planning to do... I started modernizing jaNET.

Not rewriting the idea behind it and not creating a completely different framework, but taking the original concept and bringing it to a modern .NET environment.

A new modern branch is now available on GitHub:
https://github.com/jambelnet/janet-framework/tree/modern

The core philosophy remains the same. Instruction Sets, events, built-in functions and the judo API are still there. The idea of connecting different hardware, software and services without tying the framework to a particular vendor or protocol also remains unchanged.

Underneath, however, quite a lot has changed. The framework has moved from the old .NET Framework/Mono environment to .NET 10 and can run on Windows, Linux and macOS, including ARM devices such as Raspberry Pi.

There is also a completely renewed command-line experience with command history, auto-completion, better status information and diagnostics, together with a headless mode suitable for Linux services and unattended installations.

Some new toys have appeared as well. MQTT support now makes it much easier to communicate with modern devices and ecosystems such as ESP32, Tasmota, ESPHome, Zigbee2MQTT and Home Assistant.

HTTPS support has been added to the built-in web server, sensitive configuration data can be stored encrypted and the framework can now also be embedded into modern .NET applications through the generic host architecture.

At the same time, one of the main goals of this experiment was compatibility. Old concepts should still feel familiar. Something such as `judo schedule` or an Instruction Set using `%functions%` shouldn't suddenly become something completely different just because almost ten years of software development passed by.

For those who remember Jubito, the web interface is also still there. Jubito was always essentially the application layer sitting on top of jaNET, providing a simple way to interact with Instruction Sets, schedules, events and connected devices from a browser.

You can still find the original project and its history at: https://jubito.org

For now I consider the modern branch an experimental continuation. If you are one of those people who still have an old installation running somewhere, used jaNET in the past, or simply like playing with home automation and IoT projects, feel free to clone it, build it and experiment.

I would actually be very interested to see whether somebody manages to connect an old Jubito installation with today's devices.

And who knows... if there is still some interest after all these years, perhaps proper binaries will appear again. Perhaps some more updates. Perhaps some new automations.

After all, apparently jaNET wasn't completely finished with us yet. ;)

Dynamic DNS update support for No-IP

My personal choice of a Dynamic DNS service is No-IP.

No-IP was one of DynDNS’s biggest competitors in the free dynamic DNS market, and now that DynDNS is gone they’re in a good position to snatch the crown for themselves. In fact, the two services are only one year apart in age (DynDNS launched 1998, No-IP launched 1999).
For a free account, you’ll get three subdomains on a single domain choice, but these subdomains will never expire as long as you confirm activity every 30 days. You also get port forwarding and URL forwarding, which can be useful depending on your use case. [ source ]

The story started when I received a new router from my ISP that didn't support No-IP. At that particular moment I decided to create a simple implementation of No-IP API into Jubito.

Now Jubito has a built-in function (%publicip%) that return the public IP address and a judo API that provides an update method for our hostname.

Let's proceed on how to configure your No-IP hostname.

In the settings menu, you will find a Dynamic DNS section. Click Setup, fill your hostname, username, password and save.

Next step is to create a new launcher that call the API for update. Expand Instructions menu and click on Add New Launcher. Give a name and type the action as follows:

judo noip update

We are now ready to create a schedule that updates the service in a given interval. Add new schedule and follow the screenshot, where interval sets the time of check (300000ms is 5 minutes) and action is the call we did in the previous step (launcher).

That was it. You are router independent!

Dropbox alternative for ARM Linux and Jubito integration

Once upon a time, when a netbook (Dell mini 9) with Ubuntu was my Jubito server, I used dropbox to sync my important files, such as camera video recordings of my security system. Then I migrated to a Banana Pi and currently to Banana Pro. Both derivatives of Raspberry Pi. Unfortunately dropbox has no support or plans for ARM Linux variants. A quick solution was to use ownCloud. All went well till ownCloud client stopped syncing for a number of reasons (incompatible updates/upgrades,etc).

I've made an effort to find out a solution that suits me and finally rclone came up.

Rclone is a command line program that sync files and directories to and from, and it comes with support of various storage systems (google drive, dropbox, Microsoft One Drive and more). My desirable choice is dropbox, so,  I started getting my hands wet.

I read of course that I would just have to perform the sync operation manually when I wanted things up to date but it didn't bother me in first place. I'll explain the reason afterwards.

Let's proceed to step by step process.

Download rclone to your home directory, unzip it and move forward to next paragraph, or go through command line.

cd
wget -c http://downloads.rclone.org/rclone-v1.35-linux-arm.zip
unzip rclone-v1.35-linux-arm.zip
Rename the directory for sake of simplicity
sudo mv /home/<user>/rclone-v1.35-linux-arm /home/<user>/rclone
cd rclone
Change file access permissions
sudo chmod 755 /home/<user>/rclone/rclone

Follow installation and configuration instructions. The guide use a generic name for the remote target, called remote. As I chose dropbox installation, I will call it dropbox instead.

After installation finishes try to synchronize your source with the destination. In this case the destination directory will be called jubito. Open a terminal and type:

./rclone sync /home/<user>/<source> dropbox:jubito

Time for Jubito integration. I will present three cases. Making a scheduled job, sync on-demand and trigger on event. Before we proceed we have to create invocation to rclone. We need a launcher that point to the action above.

Go to Settings->Instructions->Add New Launcher. On popup give as name "sync" and action like this:

./'/home/%user%/rclone/rclone' '/home/%user%/<source>
dropbox:jubito'

where <source> your current installation (in my case /home/john/bin), and press save. Dot slash (./) is a native implementation similar to Linux and Unix to execute a compiled program and %user% is a built-in function that gets the logged in username (alternatively, hard-coded your user). The result, if any, can be consumed by another Instruction Set. Learn more on (incomplete) Instruction Set anatomy.
This will create a pointer to the command inside AppConfig.xml which could be used at any case (scheduled, on-demand, etc).

Calling the method from CLI:

Physical location

Dropbox location

1) Scheduled

Go to Settings->Scheduler->Add New. Enter a name, select daily routine, time (e.g. 00:00) and finally sync action. This will create a schedule that runs daily at midnight and triggers the dropbox sync.

2) On-demand

From Settings->Instructions->Add New Instruction Set. Name it "dosync" or whatever you like, select from action list the sync command we created before and press add. It will generate a *sync pointer to the sync launcher. You can put some additional text if you wish. Expand Add to Dashboard and put some info (category, header) to populate a button to the dashboard list.

Now you'll be able from menu list to press and run the sync method...

 
3) Attach to other Instruction Set

First let's make an example based on DVR system by using an IP camera example. Scenario describe operations when security breach is detected (record a video and send email alerts). To extend it with dropbox synchronization, we're going to use the sync as the third action in the sequence. So, once again add the sync launcher.

<InstructionSet id="ip-cam">{ evalBool("%whereami%" == "absent"); rec sendmemail sync; ; }</InstructionSet>

If you want to use the UI and already have the ip-cam Instruction Set, please delete it and re-create it since there is no edit mode. Else, edit AppConfig.xml and change it with the line above.

However, you can create a new ip-cam with the same condition that calls only sync method like that:

<InstructionSet id="ip-cam">{ evalBool("%whereami%" == "absent"); sync; ; }</InstructionSet>

Instructions with the same name, run in order.

Upgraded Android client with location-based check-in/check-out process

Check-in/check-out refers to built-in functions (%checkin%, %checkout%) and used to change the user status from present to absent (this also could be translated as locked/unlocked or arm/disarm flag to some) and vice versa. It also raises oncheckin/oncheckout events as described in AppConfig.xml. You can bypass defaults by using custom events (e.g. mycheckin/mycheckout).

Some things in technology are meant to be deprecated, but yet, it's our responsibility to follow and make upgrades, new versions and provide new features.

I’m always looking for the best approach to perform check-in/check-out operations when I arrive or leave home. Jubito has the check-in/check-out functionality that allows us to perform a sequence of tasks depending on the event. For instance, on check-in we can turn on an assistive light, disable the camera and motion detection, checking email, etc, and inverse actions on check-out.

In the early days, I used bluetooth scan as a means to check-in when my cellphone was in range, and check-out, when it was unreachable. Drawbacks such as limitation of bluetooth range, auto-disable to save battery when phone was idle etc, didn't make it the most preferable choice.

Then I introduced the Jubito client Android app that had the capability to track my device's location, checking if I was within radius of 500m of the point I’ve set as home and do the appropriate actions as described earlier, more elegant and with more precision.

Far from perfect also. For instance, the app had to run in foreground, but besides that, Google changed the API and service eventually stop working.

The ping methodology came along, which share the same concept as the bluetooth, with some drawbacks too. Of course, it's very valuable in other kind of scenarios, but still, not good for my check-in/check-out expectations.

Lately, I found some time to study the new API and decide to modernize and upgrade the app. Along with its old functionality, now can run  as a service in background, even if the phone is locked. By far the most convenient way, especially when we don't want to be distracted by manual input on phone and be spared the extra effort.
It is a very nice feel, returning home, and this would be ready welcomes you by your preferences! Trust me on that!

Usage is straightforward. First you need to enable location from android settings. If not, and you have auto check-in/check-out checked, you will be prompted to. To take effect you need to kill the app from task manager and re-open it.

Next, from Jubito app settings, select your location by pressing the button "Get Location". On apply, selected address will be set as your home point.

Location services allow app to obtain periodic updates of the device's geographical location, and fire an application-specified event (check-in/check-out) when the device enters the proximity of a given geographical location. In our case, within 500m radius of the home location.

Before I close, let's recap what features app brings besides serving Jubito UI.

• Speech Recognition & Synthesizer. It allows to execute Instruction Sets with speech dictation and speak the result if any.
• Location & Geocoder. Reverse geocoding is the process of transforming a (latitude, longitude) coordinate into a (partial) address. That allows the app to get location data (lat/long) and reverse it to an address in order to be user friendly. By pressing the "Get Location" button you can set your home location automatically.

Closing this post with the setup of my preferences.
oncheckin, oncheckout events:

<event id="oncheckin">%unmute%; *socket-d-on; salute; judo schedule enable mailcheck</event>
<event id="oncheckout">init; judo sleep 30000; goodbye;  lights_off; judo schedule disable mailcheck; %mute%</event>

Let's take a deeper look on implementation.

All steps can be made from the UI, but in this particular article I'd like to go behind the scenes for robust understanding.

Check-in Steps

%unmute% & %mute% are also built-in functions that disable/enable the speech synthesis. *socket-d-on is a pointer to the InstructionSet that turn on an assistive light. In my case the living room right. You can also evaluate the light conditions before proceeding.

<InstructionSet id="*socket-d-on">judo serial send 000000000101010100010001</InstructionSet>
<InstructionSet id="socket-d-on" descr="Turns on living room light" shortdescr="Living room light on" header="Socket D on" categ="Automation">*socket-d-on</InstructionSet>

Salute is what it says and implements like this...

<InstructionSet id="salute">Good %salute% %user%.</InstructionSet>

%salute% function returns the part of day e.g. morning, afternoon. %user% returns the logged user. So, the above will return "Good afternoon john" and vocalized by speech-synthesis.

judo schedule enable mailcheck, enable the gmail check that is scheduled to check every 5 minutes.

Check-out Steps
init command, just speak Initiate sequence.

<InstructionSet id="init">Initiate sequence</InstructionSet>

judo sleep 30000, is a 30 seconds sleep timer, to allow me leave the place without triggering anything.

goodbye command, again a simple goodbye verbal message.

<InstructionSet id="goodbye">Good bye %user%.</InstructionSet>

lights_off command, turn off any assistive light.

<InstructionSet id="*socket-a-off">judo serial send 000000000100010101011111</InstructionSet>
<InstructionSet id="*socket-b-off">judo serial send 000000000101000101010100</InstructionSet>
<InstructionSet id="*socket-c-off">judo serial send 000000000101010001010100</InstructionSet>
<InstructionSet id="*socket-d-off">judo serial send 000000000101010100010100</InstructionSet>
<InstructionSet id="lights_off" descr="Turns off the bedroom and living room lights" shortdescr="Bedroom and living room lights off" header="Lights off" categ="Automation">*socket-a-off *socket-b-off *socket-c-off *socket-d-off</InstructionSet>

An old showcase video.

Finally, the last two are self-described. judo schedule disable mailcheck; %mute%, disable mail check and speech-synthesis.

Binaries (000000000100010101011111) in above example, control appliances and explained in this how-to.

A basic Wiki for judo API and built-in functions.

In action...

I plan to make more enhancements in future versions. For example, change the radius, add zones on specific ranges and do different tasks (at 10m open garage door), etc.

Consuming RESTful data, create widgets and more

A common way to offer server-generated data to browsers, so that it can be used in client-side JavaScript is by formatting the data as JSON, and making it accessible through its own URL. A web server that respects itself, even a lightweight like Jubito, should conformed with these standards.

Until now, the core of Jubito was loading data from the server using a HTTP GET request over jQuery.get(). The return type was a simple html. In fact, still used and applied when mode is set to html (i.e. http://localhost:8080/www/?cmd=<commands>&mode=html). By default is text, or by setting mode to text. In case of html, the only element presented is the '<br />'. The main "disadvantage" is that there is no object to handle and each command need its own call, or a manipulation of the result in case of multiple commands (e.g. splitting the breaks '<br />').

In this post, I'm going to present native APIs for consuming data from the framework and the web. That way we'll be able to customize Jubito by creating widgets and UIs.

Before we begin, let's do some REST style examples from the browser in order to understand the behavior. URI has a typical structure and addressed like this:

http://<host>:<port>/www/?cmd=<sequence of commands as parameters delimited by ampersand>&<mode=json> 
e.g.: http://localhost:8080/www/?cmd=judo%20serial%20state%&%judo%20server%20state&%whereami%&%whoami%&mode=json

Explaining the functionality of each command:
judo serial state is an API call that returns the state/status of the serial port.
judo server state is an API call that returns the state/status of the web server.
%whereami% is a built-in function that returns the user status.
%whoami% is a built-in function that returns the user login.

Parameters in API respond:
1. Pair name - A simplified name of the requested command
2. Key - Represents the requested command
3. Value - Result of the requested command

For more information about built-in functions and API please visit Wiki

Next step is to create a jQuery.ajax() request and see how to consume the response data. It provides a simple process of fetching and parsing JSON data through Ajax. An ideal and straightforward way to achieve our goal for tailor made widgets and UIs.

Sample HTML:

<!DOCTYPE html>
<html>
 <head>
  <meta charset="utf-8" />
  <title></title>
  <script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.2/jquery.min.js"></script> 
 </head>
 <body>
  <script type="text/javascript">
   $(document).ready(function() {
    getData();
   });
   
   setTimeout (function () {
    $("#data").empty();
    getData();
   }, 10000);
   
   function getData () {
    $.ajax({
     type: 'GET',
     contentType: 'application/json; charset=utf-8',
     dataType: "json",
     // requested built-in functions: %whereami% %whoami% %todayconditions% %todaylow% %todayhigh% %about%
     // request on judo API: judo server state
     // wiki https://github.com/jambelnet/janet-framework/wiki
     // uri form: <host>:<port>/www/?cmd=<sequence of commands as parameters>&<mode=json>
     url: "http://localhost:8080/www/?cmd=%whereami%&%whoami%&%todayconditions%&%todaylow%&%todayhigh%&judo%20server%20state&%about%&mode=json",
     }).done(function (data) {
      var template = 'Status: ' + data.whereami.Value + '<br/>User: ' + data.whoami.Value + '<br />';
      template += 'Conditions: ' + data.todayconditions.Value + '<br />';
      template += 'Low: ' + data.todaylow.Value + 'C&deg;<br />';
      template += 'High: ' + data.todayhigh.Value + 'C&deg;<br />';
      template += 'Server status: ' + data.judo_server_state.Value + '<br />';
      template += 'About: ' + data.about.Value + '<br />';
      
      $("#data").append(template);
     }).fail(function() {
      alert( "Oops!" );
     });
    }
  </script>
  
  <pre id="data"></pre>
 </body>
</html>

http://hilite.me/

The sample above will serve a simple page with data displayed and refreshed every 10 seconds:

Another powerful feature which you may find interesting, is the HTTP GET request supported by the API. It provides raw data from any endpoint.
judo http get <endpoint>

An example of its use can be found here on how to send and read data from RaZberry Z-Wave controller.

So, to recap, there is a lot of new things to expect, but till then, you should probably start honing your skills and finally create your own UI or a widget to the existing.

I tried to describe this post as much as possible less technical, in order to be understood by beginners.

Open Weather Map API support

Recently, Yahoo change its API and forcing everyone to use OAuth 1.0 in order to access its data. You may experience empty weather info to Jubito's front-end, as well null responses from relational functions, such %todayconditions%, etc.

I decide to implement the new functionality with openweathermap.org API as the weather data source, which I found it more simple and clear.

So, let's see what you have to do.

1) Register and get your API key

2) Expand Settings and go to Weather->Setup

3) Paste the URL you get including your APPID as a parameter, i.e.:
http://api.openweathermap.org/data/2.5/forecast/city?id=524901&units=metric&APPID={APIKEY}

You're all set. Now you can use some of the old functions, plus, the new ones for humidity and pressure. Find all the function set in GitHub wiki.

Now your dashboard should look like this:

Special thanks to Hackaday members! [ Part II ]

Indeed Hackaday. Jubito is and always will be, AWESOME!

In 2014 I was grateful to the hackaday members. Once more, I want to pay my respects to all of you out there who support Jubito.

It is the best gift I could get for 2016 and, as the note indicates, I'll wear it with pride!

Happy new year!

The Hackaday Prize: Quarterfinalists

2015 Official Hackaday Prize Entries

Project page