Smart Home Artificial Intelligence

In 2012 we started a project to allow our Home Control System to support natural language processing. This allows it to understand basic commands in English via the spoken voice or via a text interface (e.g. IM, SMS, mobile app, etc.). Our thinking back then and still is that a voice only interface is severely limiting. This approach also fits better with our user interface research and strategy.

Since then we have been extending this to essentially be a narrow Artificial Intelligence (AI) capability. This allows us to query and control every element of our smart home. Just because we designed a user interface to enable natural language interaction, it doesn't mean we would always use it. We have designed our Home Control System to be intelligent and autonomous, requiring minimal user interaction. It mostly just senses and works intelligently around us, to provide the best user experience possible and in many cases is zero-touch user experience.

What Is Artificial Intelligence?

A common definition of artificial intelligence (Wikipedia) is:
"The theory and development of computer systems able to perform tasks that normally require human intelligence".

How Is This Different?

One of the questions we get asked a lot is how this is different to the Amazon Echo, Apple Siri, Google Home and other similar services. The main differences are:

  1. These services are not particularly narrow artificial intelligence. They aim to answer pretty much any question you throw at them. This is one of the things we love about them and why we view them as complimentary to the capability that we have developed.
  2. Our system has narrow artificial intelligence that is limited to the scope of our smart home only. It can't tell you how far it is to the moon but, it is an expert on our smart home and all of the services and features within it.
  3. Our AI is an integrated part of our Home Control System and has instant access to all information, models, state and context. This is something we call whole home context and means it has all the information to hand in real-time and has no need to access 3rd party services or cloud APIs. This results in very low latency and a great user experience.
  4. Because our Home Control System and AI are essentially the same service, it can relate requests to physical objects without us having to tell the AI about everything connected in our home. It just knows about it all automatically, what it is capable of and everything about it. This means we can ask complex questions such as: "Which doors are open upstairs?" and it just knows which doors are upstairs and which ones are open, so it can answer the question instantly.
  5. Because our AI has access to all of the models, it knows what actions are possible for each object and will tell you if something isn't possible. For example it will tell you that you can't lock a door that has no smart lock on it instead of just saying: "I can't find an object called ..."
  6. Our AI is fully contextual and retains state. This means it can support a conversation thread, not just a single request and response.
  7. Because our Home Control System models people and many of our user interfaces identify the user, our AI can usually provide a personalised user experience. When this is not possible, it interacts with people as if they are unknown guests, with each "session" assumed to be a different guest.
  8. When interacting with known (trusted) individuals, the AI will use its models to enforce permissions and respond accordingly. For example, children may not be able to enable the home alarm system or change thermostat settings.
  9. These personalised experiences can also work across all of the supported user interfaces, to allow a conversation thread to move across devices and/or networks. This means you could start using a voice interface (e.g. something like and Amazon Echo) and then carry on the conversation as you leave the home using a mobile app, Twitter DM, SMS, or even email. Yes, it works outside of the home :-)
  10. There is no need to use always use a wake word. Our approach supports push-to-talk style interfaces as well, for a more efficient and reliable interaction (using a near-field microphone).
  11. Our smart home AI is an integrated feature of our smart home, inheriting all of it's capabilities. This means it can initiate conversations and not just respond to requests. It can deliver notifications for example, which can be the start of a conversation.
  12. Our @smartest_home works to our privacy requirements, keeping all our data and information private and within our home.

Design Approach

We are building our AI engine in such a way that it could work in any home, because we plan to use it in our next home. There is no 'hard coding' of features or capabilities. All the code we have written is generic in nature and applicable to any smart home. It is also driven by the simple configuration files that we use to define our smart home and everything we have connected to it and only these would need to be changed in order for it to work in any other smart home.

So far we have focussed on voice and text interaction but full multimedia interaction is now a focus of our ongoing work in this space, to enable interaction involving images, video and audio.