divan-E-Hafez: an ancient answer to a modern problem
I received this delightful message from a friend the other day:
I have been catching up on your latest work (I’ve ordered untangling the web), and I’ve been reading about your serendipity engine.
I was thinking about this in a religious context (as religion is often a place where can find long established practise that people have accepted and ‘works’ in some context).
Are you aware of the ‘divan-E-Hafez’? Iranians use this book of poetry as their inspiration for thinking about new things or making a decision. They open it at random and read the spiritual poetry on the page. They then interpret it as they like to give them a new source of direction. It is not strictly religious, as religious people will use the Quran instead of this book.
Sikhs will often do the same thing with our Holy book.
We know it doesn’t specifically answer questions, but it does provide guidance and new sources of inspiration. Could this be an ancient answer to a modern problem? ;-)
His words have pointed me in a wonderfully un-explored direction in the serendipity engine. Kat and I considered it in the first iteration of the device, but Ben and I haven’t been able to satisfactorily integrate it into mk2: the idea of divinity, karma, fate and other attributions of serendipity that involve external interceptions.
Thankfully, with the info provided by this friend, the divan-E-Hafez has been woven into the story we’re telling in The Last Bus to Serendip, the BBC Radio 4 programme about my search for the ingredients to go into a recipe for serendipity.
And in the process, our amazing researcher - Elizabeth-Ann Duffy - has discovered a virtual version, in case you’re searching…
The programme airs Thursday 5th September at 9am, and is produced by The Digital Human colleague Peter McManus.
10:01 am • 21 August 2013 • 1 note
We’re making a BBC Radio 4 special about The Serendipity Engine! Digital Human and Last Bus to Serendip (working title) Producer Peter has pointed me to this collection of videos from University of Cambridge’s Darwin lectures, and the associated book.
Tellingly, it’s called “Fortune and the Prepared Mind”, suggesting that serendipitous encounters aren’t just accident, but accident plus human.
From the blurb:
Serendipity is an appealing concept, and one which has been surprisingly influential in a great number of areas of human discovery. The essays collected in this volume provide insightful and entertaining accounts of the relationship between serendipity and knowledge, in the human and natural sciences. Written by some of the most eminent thinkers of this generation, Serendipity explores a variety of subjects, including disease, politics, scientific invention and the art of writing. This collection will fascinate and inspire a wide range of readers, highlighting the multifaceted nature of the popular, but elusive, concept of serendipity.
Darwin College Lecture Series 2008 - Serendipity
9:55 am • 6 August 2013 • 6 notes
“modeling the real world, with constraints like melting ice cream and idiosyncratic human behavior, is often where the real challenge lies”
Unhappy Truckers and Other Algorithmic Problems
An excellent article from Nautilus about the challenges of messy humans into logical processes, by way of Ben.
Some more choice quotes:
In essence, there is an answer, but there is not a solution. “By solution,” writes Cook, “we mean an algorithm, that is a step-by-step recipe for producing an optimal tour for any example we may now throw at it.”
And that solution may never come.
I should mention at this point that this article is about transportation networks across the USA, as a MacGuffin for a computer science conundrum called “the travelling salesman problem”. And here’s something interesting:
the traveling salesman problem grows considerably more complex when you actually have to think about the happiness of the salesman.
This is a lovely lesson for those people who pray to the temples of Big Data:
Powell’s biggest revelation in considering the role of humans in algorithms, though, was that humans can do it better…“We humans have funny ways of solving problems that no one’s been able to articulate,” he says.
9:22 am • 31 July 2013 • 5 notes
Additional components for a serendipity engine, designed to improve serendipity reception
In May 2013, I ran a serendipity workshop at the State Library of Victoria in Melbourne, Australia, hosted and organised by Andrew Hiskens, Manager of Learning Services at the State Library and Hamish Curry, the Education Manager at the Library.
Andrew and I first discussed the relationship between serendipity and libraries in 2011, when the engine was merely a twinkle in my eye. This was an amazing opportunity to get the perspectives of educators, librarians and other learned learning folk, and to challenge them to create a search engine out of popsicle sticks.
Andrew followed up after the event with a few notes that describe what other components he thinks would enhance an engine aimed at delivering serendipity. But first, his amazing schematic, which emphasises the need for a human being at the centre, and amplification at the end..
Now. His list of additional components:
Rationale – often serendipity is seen as a function of diversity and randomisation of experience. But the real critical issue is one of (personal) openness to receiving the serendipitous event, and processing it as precisely that. Randomisation for its own sake is just distracting and, at worst, confusing.
So here is a list of extra components designed, not to amplify the signal/s as such, but to improve key factors impacting their reception.
Part #jhk9hk4hj - Deep need extractor
For sampling desire, confusion, memory, compassion & etc to extract a deep need against which to match contexts.
Part #rtr34ik48 - Serendipity aerial
Specially designed to pick up signals which could turn out to be serendipitous (or not).
Part #n45e15 - Context modulator
As the name implies, the context modulator shifts contexts – scale, orientation, colour, inside-out-edness or outside-in-edness – in order to ambiguate the signal, so that it is familiar, but…not…
Context modulator function can be enhanced with the addition of the following bolt-on components:
Part #saeti348 - Pattern amplifier
Part #re3rere333 - Recursion attenuator – which allows pattern repetition across multiple scales
Part #th35g7g7 - Moiré pattern adjuster – for getting those weird overlay patterns you get when screening a photograph for printing, and then doing it again.
Part #rt123rr - Signal amplifier – for strengthening the thing which is the signal (but what is that?)
Part #78f9f8f - Noise amplifier – for strengthening the noise instead (but how do we know that it is not signal??)
Special human interface add-ons:
Part #3jkjkjk45j - Openness enhancer – promotes unclouded thinking, sunshine and the smell of newly-mown grass.
Part #4hi4h4jk4 - Sereneness dripper – designed to provide targeted micro-doses of sereneness.
Thank you Andrew, Hamish and the State Library for your enthusiasm, engagement and hospitality!
10:00 am • 25 July 2013 • 3 notes
What’s in the Black Box (Part 2): The Serendipity Engine algorithm
OK, yesterday I explained what social science was in the Serendipity Engine algorithm. Today, I explain the way it functions, how does it do what it does. I unveil what else is inside black box.
Remember, the Serendipity Engine was an exercise in trying to understand two things: 1) what is serendipity, and 2) how it might be “produced” by a digital technology. The first version of the engine (created with Kat Jungnickel) tackled the first question, and the second (created with Ben Hammersley) the second.
So. How does it produce its results?
The Engine uses a combination of automated and human-powered techniques to generate a personalised Serendipity Recipe.
Computers are incredible at holding lots of things in their processors at one time. This means they can cross-reference information in a way that our brains simply are unable to do. That makes them inherently better at making connections than we are, and making connections between things is a prerequisite for serendipity.
But at the same time, computers aren’t very good at assessing human qualities like physical attractiveness or creativity. To say whether someone has these traits require human judgement, which is why the Engine is also made up of human components.
How does it work?
The forms you complete feed the Engine with two things:
A) keywords that identify what should be relevant and valuable to you
B) your level of Serendipitousness, measured across seven scales
Step 1: The Paper Form
The information fed to the Engine on a paper form contributes to your personal Serindipitousness rating.
You are asked to do three things: draw a circle, create a drawing from a squiggle and photograph it with your face.
The photograph of your face, your circle and your drawing are sent to a system called Mechanical Turk. Mechanical Turk is an automated, human-powered “computer” run by Amazon that farms out the projects that can’t be automated by a machine to people who, for a small sum, take a look at what needs doing and complete the process. In this case, someone somewhere in the world was paid $1.50 to look at the photograph you submitted and is asked to assess it in four ways:
1) The circle you draw is compared with a Japanese ensō, a symbol that represents universality, openness and creativity
2) The drawing you make based on the quiggle is rated for its creativity
3) Your face is assessed for its physical attractiveness
4) Your face is rated for your psychological well-being
Step 2: The Online Form
The information you provided the Engine on the online form is used to identify the keywords that are part of your personalised Serendipity Recipe. The form splits into three sections, with three different types of information:
1) The first four questions authenticate who you are by cross-referencing what you have told the Engine about your basic demographic information. This is done by using a Google search of your name and checking the search results with other details (house number, post code, age, etc). After identifying you in Google, the Engine extracts keywords from the top three search results that define who you are in public.
2) Your social media handles are used to identify what is most relevant to you right now, by extracting nouns from your 10 most recent status updates on Twitter, Facebook and/or Google+.
3) The remaining questions are used to extract keywords about you that online databases are very unlikely to have on record. Questions about your favourite subject at school or your Desert Island Disc, or even the food you’d eat tonight if today was your last day on Earth require a much more nuanced level of self-assessment and personal expertise than is catered for in online profiles. This is also the information that can’t be collected based on your browsing behaviour.
In other words, what you listed as the most influential film in your life may (or may not) be different from the film you’ve watched the most, or rented from Netflix most often, or even what you listed on a social networking site as your favourite film. And your parents’ specialist subjects may be completely different from what they do or did for a living. The Engine extracts keywords from tags associated with each from open data sources for each of these categories.
Step 3: Your Serendipity Assessment
Finally, there are 23 questions in the Serendipity Assessment section of the Engine. In the physical version, they’re inside a suitcase and made of knobs, switches and dials. The questions form seven scales, each of which has been identified in the research literature as important predictors of serendipity. I described the scales yesterday. The outcome of your answers to these questions, combined with the responses from the Mechanical Turk, is a measure of your personalised Serendipitousness.
There are also two questions in this section that extract even more keywords: What is your profession? What is the nature of your business?
How’s my personalised Serendipity Recipe determined?
By completing the Paper Form, the Online Form and the Serendipity Assessment, you provide all the information the Engine needs to deliver a personalised Serendipity Recipe.
First up, the words.
5-10 keywords based on your responses to the keyword questions in the Online Form and the Serendipity Assessment are randomly selected from the long list of keywords you gave the Engine access to, whether you told it directly, or it extrapolated based on its automated searching system. These keywords are then put through a process of filtration determined by your Serendipitousness score.
Your level of Serendipitousness is based on your answers to the questions in the Serendipity Assessment and to the Mechanical Turk’s responses. These are aggregated and weighted to produce your Serendipitousness Score.
The scales are weighted differently according to how important each is in predicting whether someone will think something is serendipitous or not. This is a difficult combination of things, but briefly, it’s about how likely it is that you’ll be able to connect the random keywords in your personalised Serendipity Recipe, and how likely you are to think that those connections could be valuable to you.
Creativity, Attention and HeadRAM are the two most important scales in the Assessment, and so they’re weighted the most heavily. The other scales - Social Support, Physical Well-Being, Psychological Well-Being and Grit - are also important, but for the purposes of this Engine, they are equally weighted.
See more about the scales and their weightings.
You said it’s filtered? What does that mean?
The randomly selected keywords are filtered through Google Translate. If you have a low level of Serendipitousness, the keywords will be translated only once from English to German and back to English. If you have a high score in the Serendipitousness Assessment, your keywords will be translated up to 5 times.
Why do that?
The Engine creates relevant randomness. It takes things that you are likely to pay attention to and puts a new spin on them.
In order to the Engine to work for as many people as possible, it has to cater to the different abilities of the people who use it. Serendipity is a very relative thing, based on where you are in the world, what time it is, how you feel that day, what resources you have access to, what political regime you live in, and which culture you’re from. Some of these can vary day-by-day, and even hour-by hour. So to cope with the constantly moving target of your own Serendipitousness, the Engine works for everyone.
The more filtered the keywords are, the less like the keywords you provided. Which means the more connections you might make that are tangential to who you are and what you know. The less filtered the keywords are, the more like the ones you put in. In other words, the Engine makes you work a bit harder to find the connections, have the insight and see the value if you are more serendipitous, and less difficult if you aren’t.
Is that it?
Almost. There is one more thing. Serendipity is often associated with a flash of insight out of the blue: when walking the dog, taking a shower, having a cup of tea. And so the Engine suggests three contexts to guide you to where you should consider the words, what you should be doing, and who you should be with.
Yup. Predicting serendipity. It’s that easy.
* Thanks to Nominet Trust and Google for their support in this research.
10:00 am • 24 July 2013 • 3 notes
The Serendipity Engine Algorithm: What’s in the Black Box (Part 1)
So we created this machine - this Heath Robinson device (as Prof John Naughton described it in The Observer on Sunday) - but how does it do what it does? What’s inside the black box?
The Serendipity Engine was an exercise in trying to understand two things: 1) what is serendipity, and 2) how it can be “produced” by a digital technology. The first version of the engine (created with Kat Jungnickel) tackled the first question, and the second (created with Ben Hammersley) the second.
Here’s the answer to how the first fit into the second:
Scales, weightings and their sources
Your Serendipitousness is based on the answers to questions that fit into seven different scales. Some of the questions are considered more “important” for predicting attributions of serendipity (the likelihood you’ll have the insight to make the connections and that you’ll consider the connections valuable), and some of the scales are considered more “important”.
These scales are based on psychological tests. As such - and just like a computer - they are unable to capture everything about the thing they’re trying to measure. They can’t take into account all of the various things that should also be taken into consideration.
If you are high in social support, you’re probably in a tight-knit group of friends and family whom you rely on. So the decisions you make that seem out of character will likely to be supported by your friends and family. Because of your demographic make-up, doors may open for you more easily than for people with a lower score. But you may not have as much access to new information as someone with a lower social support scale. On the other hand, if you’re low in social support, you are probably on the periphery of lots of different networks, and so you’ll have lots more information at your fingertips and you’ll be more innovative because you’ll have less to lose if you decide to do something out of character.
Creativity (x 2 weighting)
How creative will you be in making connections between things that seem to have no apparent connections? If you’re more creative, you’re more likely to see affiliations between things, and it’s more likely that they’ll be new and unexpected. If you’re low in creativity, you will still see connections, but they’re probably more literal.
If you’re body is too busy trying to keep itself in optimal condition, it won’t have the energy or resources to pay attention to things or to make connections.
HeadRAM (x 2 weighting)
This scale measures how much you can keep in your head at one time. If you can keep a lot in there, you will be able to access it more easily, and therefore will be able to make lots of connections.
Attention (x 2 weighting)
How much attention can you give to your surroundings, and are you easily distracted? If you don’t see or can’t keep your eye on the little random confluences that could lead to a serendipitous discovery, you’ll miss ‘em.
Access to Knowledge
If you have a high score in this scale, you have a very wide range of knowledge at your fingertips. It may not be your own knowledge: are the areas of your parents’ expertise very different from your own? You’ll still have access to that, even if it’s just half-remembered titbits from your childhood. With high access to knowledge, you’ll be able to see more connections between things, and have a diversity of information to help realise their value.
Grit is a measure of your tenacity - to solve problems, to see something through to the end. This can be particularly useful in predicting whether you think something is serendipitous or not if you’re low in Social Support and the thing you want to do is outside your comfort zone: if you’re high in grit, you’ll do it anyways.
Tomorrow, I’ll explain how this is used to deliver your personalised “serendipity recipe”.
* Thanks to Nominet Trust and Google for their support in this research.
11:22 am • 23 July 2013 • 13 notes