The consequences of dropping the ball in digital engagement - The ABS and Australian Census 2016

Next week Australia will be holding its 17th national census (since 1911), led by the Australian Bureau of Statistics, which is itself celebrating its 110th anniversary as an agency (albeit with a name change midway).

This is an auspicious occasion for another reason. While it has been possible to complete the census online in both 2011 and 2006, when the ABS first trialled an online completion system - 2016 will mark the first occasion when the ABS expects a majority of households to complete their census surveys online.

In fact, Duncan Young, head of the 2016 Census process, is on record stating that the ABS expect 65% - two-thirds - of households to complete the Australian Census online, rather than in paper form.

This is a fantastic achievement and speaks highly to the ABS's commitment to quality data collection and maintaining a forward-facing approach to trialling and adopting new technologies.

This commitment has also been typified by the ABS BetaWorks Blog (sadly now defunct), ABS CodePlay (sadly not repeated) and the work the ABS has done to expose data in open and machine-readable formats, including ABS.Stat and APIs such as for the Population Clock.

Data collected by the ABS, particularly via the Australian Census, underpins an enormous amount of evidence-based decisions made by all levels of Australian government, as well as by companies who access the information to guide their commercial decisions.

The census is also an enormous undertaking. To quote Wikipedia quoting the 2011 Census site, "the 2011 Census was the largest logistical peacetime operation ever undertaken in Australia, employing over 43,000 field staff to ensure approximately 14.2 million forms were delivered to 9.8 million households." The cost was $440 million.

That makes the census a prime target for budget cuts - with the idea of reducing the frequency of the Australian Census to every ten years, or reducing its complexity, thrown around last year before being dropped.

The impact of not having regularly collected census data, collected in a compulsory manner from all households, can be hard for Australians to imagine.

However in countries like Lebanon, which hasn't had a census since 1932, the lack of accurate data leads to opinion-based government decision-making, which is generally viewed as a poor alternative to fact-based policy decisions.

The need for compulsory collection of census data was highlighted by the decision by the former Canadian government to make their long-form census voluntary in 2011, resulting in a massive drop in participation and corresponding degradation of data quality.

Called "a disaster for policy makers", unfortunately it suited the Canadian government of the day to not have accurate data in order to provide them greater room for making ideological decisions, rather than decisions that were based on facts. The net result was a drop in participation from 95% to 68%, a more expensive Census process (due to increased mailout of forms to prompt engagement), the resignation of several of the most experienced and competent senior officers in Canada's statistical agency, ongoing issues for national, provincial and local Canadian governments in identifying disadvantage, population numbers, statistical population changes and reduced capability for companies to make appropriate commercial decisions without investing in further expensive research.

The current Canadian government reinstated the compulsory long-form census, which completed collection in May this year.

So regular compulsory censuses are a BIG DEAL for a nation, and Australia has a very strong statistical foundation to build on.

The ABS has also demonstrated leadership in how it has marketed and communicated past Australian Censuses. In particular in 2011 the ABS demonstrated global leadership in the use of digital channels and tools to promote the importance of the Census and lift participation.

Through quirky best practice engagement on Twitter and Facebook, which made the Australian Census front-page news for all the right reasons, the development of an interactive online service allowing people to 'place' themselves within Australia, and a mobile game which allowed people (particularly kids) to see how census data was used in civic decision-making, the ABS knocked it out of the park in terms of its communication strategy and implementation.

That's a fantastic base for the ABS to build from. I think a number of people were expecting the same, or better, engagement from the ABS in 2016.

Alas, it was not to be. In 2016 little of the previous engagement brilliance is evident from the ABS.

While the ABS has repeated a level of their communication via Twitter, it's basically a shadowy repeat of their 2011 strategy - as though new management said "repeat the good stuff from five years ago, but don't update anything or take any risks".

The ABS is also remaining stalwart and largely silent in the face of several decisions which have left census collection exposed.

Their online service has been exposed as using an older and less secure security standard in order to support older browsers, rather than taking an approach which warns people and encourages them to upgrade to a more secure technology.

For non-technical people, an analogy would be the police waving past someone without headlights on a dark night onto a crowded and unlit highway in order to not slow down the traffic flow.

On another front, the ABS is confronting a surge of privacy concerns around its decision to keep names and other personal details connected to census data for at least four years. Taken without consultation with the public, this decision has raised alarm bells with privacy advocates and organisations such as Electronic Frontiers Australia, as well as with former senior officials of the ABS.

While the ABS has been fighting back to some degree, they've not really addressed the concerns in an effective way.

#Censusfail is continuing to grow as a hashtag, with a number of people considering ways to circumvent responding to the census, avoiding providing personal information or considering providing false information.

Should enough people take one of these steps it would reduce the value of the census to Australia.

I must admit that I've also become concerned about the ABS's approach, and unconvinced by the ABS's engagement on this front to-date.

I totally support and value the ABS as an organisation, and all the people that work there - however they are burning much of the goodwill they established in 2011 and potentially devaluing the census, and hurting all Australian governments through their lack of effective engagement on the issues above.

The worst thing for me is that the ABS has been a shining light in Australian government. The organisation has consistently been a leader in open data and the use of digital and social media to engage with the public.

This is important not simply for the egos of the leadership at the ABS, but is essential for good governance and effective commercial decision-making in Australia. The ABS's success serves all of us - and its failure would hurt us all.

I hope the ABS recovers from this and Australia continues to be well-served by the statistics the organisation collects.

However it would have been far better for the ABS, and all Australia, if the ABS hadn't put itself in this position of needing to recover at all.