2017, the summary
This post is part of a chain of yearly summaries of noteworthy self-education and side-project work. I'm publishing these overviews to provide pointers to interesting resources, to show I'm serious about continuous self-improvement, and to inspire others to have fun with new technologies.
Previous year: 2016 <---> Next year: 2018
2017
This year was characterised by my move to Vancouver, and start at Amazon. Moving to a different continent obviously comes with some downtime in side-projects while settling in a new home, starting a new job, and adjusting to the new life, but can also inspire to change habits and look at new resources.
Changing to corporate life coming from startups turned out to be quite a step sideways. There was the downtime while learning the many internal tools and unmappably-big network of services; there was adaptation to corporate culture (more resources, less enthusiasm and idealism, more politics, less quirky hacking); and there were changes while transforming in a good Vancouverite (more trips to beautiful nature and nearby ski areas, less cultural activities or drinking at canal-side terraces). Let's get going with the few side-projects that happened, and a bit of reading.
MXNet intro
I followed the introductory material for MXNet called The Straight Dope. MXNet being the framework of choice for Amazon, I was curious to dive into this modern Deep Learning framework. Despite not being the most popular framework, I found MXNet - together with its 'keras-like' interface called Gluon - to be easy to use and pretty complete. The materials included examples in various applications (NLP, computer vision, recommender systems), as well as advanced topics such as distributed training. The ndarray library is a nice drop-in replacement for numpy, and has an efficient implementation using computation graphs, automatic differentiation of about any function, caching of intermediate results, and GPU support, making it quite useful beyond just neural networks. MXNet is definitely a library I'll be exploring further.
Max / MSP
Mentioned by a friend years ago, I took some time to experiment with Max/MSP. Max is a node-based, interactive visual programming tool optimised for music and multimedia projects. The bad news is it's Mac/Windows only, and paid (1 week trial). Nonetheless, Max has a long history starting in the 80s, and its interaction model inspires new ways of programming on an advanced level I haven't seen elsewhere, with support for surprisingly complex programs. Coding happens by way of patching together components sending signals, mimicking analogue synthesisers and allowing visual programming. It is possible and encouraged to create programs interactively, by changing running sketches and interacting with the many available interactive widgets that control the flow of signals and functionality of the blocks.
Max is very good at audio and video processing, with both high-level and low-level blocks available and highly optimised signal processing backends, but can also be used for arbitrary programs as long as they can be mapped to data flows. It supports abstraction (nested sub-patches usable as new nodes), extensibility (JavaScript or Java blocks, coded and changed at runtime), creation of flexible user interfaces with interactive UI elements (controllable and changeable at runtime), meta-programming (creating/changing sketches at runtime), built-in support for regular events, continuous signals, and efficient matrix manipulations, built-in blocks for many protocols (TCP, serial, midi, mics, file formats, OpenGL), and runnable help example pages that are editable patches themselves.
I followed this awesome course by Matt Wright, who goes over many basic and more advanced features while giving cool interactive demos, building synthesisers, OpenGL VJ software, simple computer vision patches, and protocol convertors as we go.
Similar open-source tools are Pure Data (PD), Node-RED, Scratch, Processing, and SuperCollider. Popular commercial node-based environments that seem similar are Ableton (which comes with Max support) and TouchDesigner, but I didn't have time to look into those yet. See also my overview of Creative Coding tools.
Node-RED
Node-RED is a visual flow-based programming environment allowing quick wiring of drag-and-drop nodes (predefined or custom code blocks), processing flows of messages, and connecting Internet of Things elements, devices, protocols, and APIs. Contrary to Max/MSP, it is open-source, and comes pre-installed with Raspbian on Raspberry Pi's. It is not as advanced, though, and there are no visual or interactive widgets or blocks (a big shortcoming!). Nonetheless, there are many nodes available, including HTTP, web-sockets, sockets, MQTT, OLA DMX, debugging, Alexa, IFTTT, Raspberry Pi hats, twitter, smartphone notifications, popular databases and queue systems, file handling, and a wealth of third-party Node packages. Custom JavaScript nodes can be written on the fly, or packaged and distributed back to the community.
Node-RED seems very useful for quick prototyping, changing logic on the fly. Its flow-based programming model maps well to typical IoT applications, which usually involve events that trigger processing, but is also useful in many other applications. It's similar to IFTTT, but allowing for much more advanced data flows.
Node-RED quickly became my go-to tool for not only home automation rules, but also as sketchbook for quick mini-apps, or locally hosted "server-less" or devops-free scripts, that can be built instantly from any device by accessing the Node-RED web interface. In a few days, I clicked together sketches to hook up physical remotes to my home automation system, to add Alexa to control the same system, enabling flows to use Google Home's text2speech functionality, created a mini HTTP website for viewing its logs including web-socket stream, made flows for sending notifications for sudden stock changes and missing backups and increased negative sentiment on Twitter, created a simple stand-alone blog, made a quick dashboard for viewing historical stock prices, and added streaming those graphs to internet-connected LCD screens.
NodeMCU
Arduino is a great platform for quick hardware prototypes, but lacks (cheap) support for WiFi. While making things talk to each other, it would be great to have the ability for internet connections built-in. The ESP8266 chip and full NodeMCU development board has this functionality and is (almost) completely compatible with the Arduino. You can buy them from Aliexpress for just $2.50 each and can be programmed with the regular Arduino IDE, ignoring the Lua support while supporting most third-party Arduino C++ libraries out of the box. I got some issues with a few hardware libraries (e.g., NeoFire library), but overall most Arduino sketches seem to work out of the box.
The NodeMCU platform has become my default microcontroller pretty quickly and forms a great platform together with Node-RED and some custom nodes. I standardised on a sketch with my custom written REST API, and built a bunch of lamps (some just dimmable, some full RGB LED strips) and LCD screens now spread around the apartment. It's also trivial to create nodes with a bidirectional open stream of messages using MQTT, which I used for a sensor node at the door.
Recommended Readings
Besides reading books in what some tell me to be the "legacy" format - physical books, OH MY - I tried audio books this year. Despite being skeptical at first, I found listening to books quite enjoyable. It's great for listening after going to bed or while on the go (when reading would be impractical), you can listen together if you have shared interest, and you can increase the speed. It doesn't work for very technical books - with diagrams and code examples - but other than that works great for both fiction and non-fiction. Listening to books also works at moments you might otherwise be tempted to watch a series.
Here are my favourites from the books I read this year.
Software Reliability Engineering - various Google employees
An extensive collection of essays on how Google runs large-scale production systems. The selected articles are loosely related, and can be read selectively, but I read it all. It touches upon risk management (give development teams an 'error budget'), balancing DevOps and infrastructure development, OnCall and incident handling at Google scale, cool probabilistic load balancing, overload, and cascading failure algorithms, distributed batch jobs, descriptions of their development and deployment systems, and many anecdotes on failures (and successes) from the organisation within Google called Software Reliability Engineering.
Being a developer at Amazon, I found it amusing to see how Google seems to have arrived at similar solutions for many problems, and might have found better solutions for others (I assume there is some publication bias at play here). The book is also available online and worth looking through for good practices.
Peopleware - DeMarco & Lister
A classic book from the 80s (since slightly updated) with timeless advice on how to run software projects. Its main insight is that management is not about solving technical issues, but sociological problems. It's an illusion to think software development is high-tech; we're mostly applying existing knowledge and are otherwise spending most of our time communicating (via code, emails, or meetings). As a manager, don't make people work, but make it possible for people to work. Peopleware also describes many good practices related to hiring and ramp-up, preventing burn-out, healthy and distraction-minimised offices, team building ('spaghetti dinners'), and org-level changes.
Although the book is biased towards large corporates, most chapters were rather relatable as a previous startup founder.
Power of Habit - Charles Duhigg
Our brains consist of different areas that determine different types of behaviour. We prefer to think our high-level, cognitive brain determines our behaviour, but much (40%) of our days are spent executing routines, led by the older, less conscious part of our brain. It's the part where habits exist and, with some practice, can be changed to help us achieve long-standing goals that need regular steps. This books includes tips to both change existing bad habits, and to develop completely new ones. The trick is to find the cue that leads to (thoughtlessly) executing a routine because we crave for some reward, and learn to replace the routine with something we actually want, leading to a reward that's aligned with our goals. Examples include reducing obvious bad habits (addictions), but also learning completely new habits (a new sport, regular writing), as long as it can be triggered by some cue and leads to a pleasurable reward to crave for next round. Similar arguments are made for habits of organisations and of societies.
Algorithms to Live By - Christian & Griffiths
A rather enjoyable journey across algorithms and math applied to everyday human situations. This geeky book is very readable (or listenable) and gives surprising insights into mathematically proven optimal strategies for very human dilemmas like finding your dream home or life partner, the best way to sort your bookshelf, trying new restaurants, and family schedule optimisation challenges. Written for a non-technical audience, you'll still find yourself chuckle when the authors apply your favourite algorithm on a very different context.
Chaos Monkeys - Antonio Martinez
Antonio describes his epic journey from the investment world, to founding a startup, selling it, and working at Facebook. This 'boy's dream' story is full of pointy comparisons and funny anecdotes, and the author doesn't shy away from verbalising his opinions. I laughed with him when reading relatable episodes in the crazy world of startups, and cried about the slower and bureaucratic corporate world while he was transitioning into one of the Big Five.
Sapiens + Homo Deus -Yuval Noah Harari
Two fascinating books on the history (Sapiens) and future (Homo Deus) of the human race. From the very beginnings of this once insignificant species, to the development of modern cognition, and important inventions including the agricultural revolution (not necessarily beneficial for the individual but helped the population grow), writing and math symbols, and most importantly the shared myths (money, religions and other ideologies, empires and countries, and companies) that enabled us to have common cultures in societies larger than ever seen before. The author gives a convincing framework for thinking about the accelerating speed of developments and increase of productivity, which started with the scientific revolution and was accelerated by the industrial revolution - spreading with European expeditions (and looting), and (like it or not) with the growth of the capitalist ideology, enabling a vicious circle of belief in progress and loans/investments to make that very progress possible.
The second book, which is more speculative, deals with possible paths in the future. It dives deeper into how the human race ended up in its current stable state with relatively few wars, plagues, or famine, and proposes directions on where to go from here. Contrary to most books, it does not dive into technical possibilities; instead, it talks about the social implications given a wealth of historical context. I found it enlightening to have the, often wild, utopian views in the future framed within underlying societal developments from the last few thousand years. Although I found the first book more interesting and better organised overall, this sequel is worth reading if only for the part on the transition from God-centric, via human-centric, towards a data-centric (dataism) society.
Digital Gold - Nathaniel Popper
2017 was the hottest year so far for cryptocurrencies, with the hype reaching front-pages on a daily basis and flowing into many christmas family dinners. While a book one year old might seem already outdated for cryptocurrency standards, it is instead a fascinating read on the inception and early growth of the distributed digital money called Bitcoin, with the stories of the eccentric characters that made it possible. From the very first emails with Satoshi Nakamoto and the idealistic cypherpunks til the awakening of anonymous marketplaces, Bitcoin entrepreneurs, and regulators trying to grasp this new phenomenon, this coherent history of the first successful cryptocurrency reads like fiction. You won't find any math in this book (read Satoshi's paper instead), but you will find the story of very human characters in a remarkable digital revolution, whether it will succeed or not.