Sunday, October 29, 2017

29 Oct - ESP-8266 + network + neopixels

Quick links for the Notifier widget.

Tested, working - Neopixel library (with network working ok): NeoPixelBus
(and works with NeoPixels and older WS2811's in 5mm LED style packaging)

I didn't have much success with the Adafruit NeoPixel library on the ESP-01; odd since I'm sure it's worked before.

Grab this and restart the Arduino environment: SPIFFS + Arduino IDE; I used ver 0.3.0. In mac OS go to, 'Show Package Contents', navigate to Java, create 'tools' if it doesn't exist, and copy the unzipped download there ( the complete path will be Java/tools/ESP8266FFS/tool/esp8266fs.jar )

Small Notes:

- Tx is pin 1.
  • I use it to measure the ambient level, as on/off, of an LED in a night light.

- Rx is Arduino pin 3 / GPIO 3. 
  • It's tied to the high speed DMA USI? i.e. can sort of draw NTSC video. I use it wit the NeoPixelBus library as the output pin for a couple of WS2811's to display status info.
- Docs on ESP8266WebServer are at: and the decision to use it was based on ready access to GET / POST data and not having to re-write HTTP handling.

- Writing monolithic servers (one TCP port, with services presented within the packet payload) is dumb. I'm mapping out the services now and better to implement as a series of libraries rather than giant bowls of spaghetti.

Thursday, October 19, 2017

Randomness (to 17 Oct 2017)

Not too much on the solar tracker, directly. But:

1. Side project: Installed some reed switch door sensors yesterday. Significant note: It's way WAY easier to get the magnets and sensors lined up when you use expanded PVC board, and heat it up to flex it into position.

2. Reduced the complexity of wifi messaging for IoT devices to a basic 8 types:

  1. Condition
  2. Configuration
  3. Command
  4. Communication
Each has a requestor state and response state, so that makes 8. This allows for all combinations of state-machine transitions and responses that make sense for both buffered and real-time scenarios.

3. Finally got the garlic planted. Although this appears to have little involvement with electronics, it will later when I start instrumenting the garden, and how to deploy that in a weather sealed fashion. Short version: ABS pipe + fittings.

Sunday, October 15, 2017

Multiport Server in Go...

Today was a little windy for outdoor work on the solar panel tracker, so I updated the Go code to include a more graceful exit:

  • Logging server listens to quit chan
  • Time server listens to quit chan
  • Quit server closes quit chan
  • All servers quit, which causes defer to close all db & net connections
  • Main quits and exits to os.

This way I can telnet to port 6660 and kill all the services in one swoop, and know that the db connection was also gracefully closed.

Friday, October 13, 2017

Quick note on generic logging packet formats

Before I forget - I now have a generic logger running on a Mac, written in Go.

It listens on a specified IP address + TCP port number, and if the format is ok, writes it to a mySQL table. I can send from any wifi enable project, like from an ESP-01, for example.

The format (today) is composed of pairs of values, to 24 bytes long (so, 12 pairs). The two parameters are called opr (operator) and prm (parameter). The byte layout is:

0,1Source Network (opr) could be RF, WiFi, wired I2C, SPI, etc
Source Node ID (prm) the node ID on that network
2,3Destination Network (opr)
Destination Node ID (prm)
4,5Payload Type (opr) - actually - split into 16 media types + 16 payload types
Payload Version (prm) - for any media+payload, also a version to support mixed version environments to allow for graceful / rolling upgrades of equipment firmware
6,7MAC - Message Authentication / Challenge - Lo (opr) : 16 bits of message authentication
MAC - Message Authentication / Challenge - Hi (prm)

The choice of length might seem arbitrary (and REALLY short) but it's to allow for nodes connected via RF (i.e. a Nordic radio, like the nRF24l01) to send at an optimal packet size. Some of the tests I've seen but not yet replicated suggest that good performance can be had with 24 byte packets, but loss/corruption seems to be more of an issue at 32 bytes.

To maintain flexibility for true WiFi connections it shouldn't be too much of a stretch to specify a different Payload Type which could extend the packet to hundreds of byte pairs, probably 1k is a good limit if the ESP-8266 is going to have to buffer any of this.

I specify the networks and node ID's separately from the IP info because it's likely that some nodes are only connected via I2C/UART or IR links. In this way intermediate nodes can act as routers to forward the packets, hopefully to some kind of hub or handler.

The data section, for logging, can be viewed two ways, depending on log type.

The first version is to send register values; so the opr is the register #, and the prm is that registers value. This means I don't have to send sequential register space, but it's twice as slow to send. It does mean I can send ANY register space in any order, however.

The other version is to send specific formats, where every byte from 8..23 has a specific meaning for that node. It's less flexible, but might be quicker for sending mid-rate data blocks, like compressed audio.

I'm leaving encryption out of this spec. Adding it would mean simply a prior encryption exchange, and then probably encrypting only the data bytes. I suspect I'll use offline updates of one-time pad tables - but that's another post.

The last note (and probably kinda important) is why the bytes are labeled as OPERATOR and PARAMETER. Not written (quite this way, anyway) is an interpreter that basically looks like assembly language, and consumes ONLY two byte instructions; one byte for opcode, on byte for parameter (literal, offset, etc).

So this means that while I can use the transport right away for logging, it can also carry command and configuration instructions later, in different payload types.

Software Notes

I was building a 'quick and dirty' web crawler to pull some information into a single page; decided to give phantomJS a spin. It works - sort of. For some of the sites that require some AJAXness or hiding certain obstructive divs it was starting to turn into diminishing returns. I still have to look into casper as an improvement. I might just switch it to Go...

Speaking of which - I also did some work on the solar tracker; before I put the electronics outdoors I wanted to see if I could improve the wireless aspect to more than just responding to http requests with an ESP-8266. For this I'd need a server indoors. My first thought was to run it off of the notifier node,  but it needs an upgrade to actually have the ESP wireless link connected LESS often, to allow more use of power for lighting instead of WiFi.

It turns out that a single page of Go is enough to accept a tcp connection, connect to a mysql table and dump information from a wireless node. I toyed with the idea of doing visualization + network in Processing, but it struck me that I'd probably just connect to the db for visualization requests remotely anyway. Once I'm happy (happier?) I'll post some docs here.

Last up, I think I'm going to add a temporary fabric cover to another project, to keep ice & snow off of it ('software', get it?). I can't fully enclose it, but maybe I can keep the wet stuff from soaking it from above.

To that end I found a snap kit that looks like it functions pretty much like the grommet kit. I'd use only velcro, but the winds will be in excess of 100 km/h so I'd like to try something a bit more positive-locking.

13 Oct 2017 - Mk 1 Tool Cart

Tool Cart - Mark I

Yes, generally inspired by @donttrythis... (thx Adam!). Probably should open source the design...

Main features:

  • Wheeled!
  • 5' tall at the back for a 2' x 4' pegboard
  • 3 main shelves, holds 3x parts sorters, each with 4 drawers
  • 15 mini organizer shelves, 5.5" wide (10.5" deep)
  • Upper 'desk' area for flexible tool storage/access;
    • shown with foamcore mockup stand for magnetically linkable 'tool arena storage'
  • Top surface for beverages, I guess
  • Jurassic-era beverage opener above drill & hot air holsters on right side
  • Mounted power bar for local outlets
  • Left side not defined yet - probably shelves for common glues, paints, tools
  • 1 mid level drawer @ stool working level:
    • Holds 1 of 4 different multi task trays on top of drawer (from legacy Ikea)
    • Holds parts organizer drawers inside (from parts bins)
  • Illuminated via LED's powered by rechargeable AA's above & under mid level drawer
  • Entirely built of wood I had lying around. Some of the copper preserved wood was actually stepping pads in our garden, and had been outdoors for 10+ years.


Other Notes:

This isn't done; this is Mark I.

I have no idea what I need on the left side; shelves, long tool storage, long materials storage... dunno.
Also, the tool arena on the top isn't defined. I suspect I'll use the white foam core mockup to build several, in thin plywood, and then drill / cut the tops to fit various task oriented sets of tools so I can swap them out as needed.

I got lucky with the white plastic organizer trays; partway through the build I realized I had these 'dying in drawers' and should really get them populated with common-task tools and consumables. It turned out that the trays fit on top of the drawer box perfectly.

The drawer itself is an attempt to address _directly_ what Adam had to build for his Sortimo shelves -  a standalone rolling cart. It was $5 for two 12" full extension bearing sliders and some scrap wood. This drawer is built-in, since I have way less floorspace than his cave has (about 250 sq ft, on a good day).

The LED lights were done on a lark. I had these from Michaels - $5 or 10 on sale - so why not? It chases the shadows away when the lighting isn't ideal.

Things for Mark II:

Wider from front-to-back. It's right on the edge of being tippy. It's not tippy now, but that's not to say it won't be tippy if someone falls and grabs onto it. Probably centre the casters at 16" deep (they are about 13" deep x 26" wide now)

MOAR lights. No kidding. The current lights are useful, but should be 2x as bright. Maybe 3x.
The original plan also called for a permanent 2x USB outlet from a LiPo battery; one for 'whatever' and one for an Ikea USB 1W flexi lamp. Recharged when powered from 110V; otherwise the cart is ok on battery for a week or so.

Thursday, October 5, 2017

Not-Status-Snapshot, 5 Oct 2017

After getting the enclosure over the mast and drive sections I took a break from this project and continued on getting everything squared away in the shop.

The more time I spent tidying up and finding tools and materials, the more I realized I needed to use the space I had more efficiently, so I decided to whip up a rolling tool & parts cart.

Aside from some details the cart is done and works well. I'll post some images and dimensions later. The only disappointment was that the local store was out of stock for the storage inserts, so that part will have to wait.

To Do on the cart:
  • Slide out shelf for tray use
  • Affix the lighting inside the frame so I can see the bins better
  • Finish / attach the holsters for the hot air gun, drill, and glue guns.
  • No ideas yet for the left side - could be storage or blank for now
  • Found the bottle opener, will attach that plus a magnet.
I also moved the machinist files from the electronics work area to the dust enclosure, but adding an external rack to drop them in. I was never happy about having metal shavings so close to surface mount electronics, so now it's a non-issue.

Tomorrow I should be able to run the wiring and the connectors for the solar controller:
  • Wiring:
    • Power (mid gauge, +12V and GND)
    • Comms (just two wires, light gauge for I2C to the power box logic)
    • Terminate on a 4 pin trailer connector
  • Add connector for azimuth feedback pot
    • This runs to A6 and A7, so I'll need to double up on +5V and GND from the connector
    • Maybe a daughterboard? Polarized 4 pin JST? Not sure
  • Add spring to azimuth feedback tensioning