Developing the world's first transnational open beneficial ownership register: A brief history of the Open Ownership Register

Iteration

Over the seven years since its launch, the OO Register grew to encompass data explaining more than 33 million BO relationships between over 11 million companies and 10 million beneficial owners. This data was republished from Armenia, Denmark, Slovakia, and the UK, in line with BODS, and updated each month. Thousands of users visited the site each month, arriving mainly via Google searches, and Open Ownership explored a wide variety of ways that international, structured data could be put to work to generate insights into BO networks.

A timeline for the development of the OO Register is outlined below, plotting the addition of new data sources and features alongside the lessons learned from this work.

Adding Ukraine data (January 2018)

In May 2017, Ukraine became the first country to commit to integrating its BO data into the OO Register as part of the Open Ownership pilot programme. This achievement was made possible thanks to the tireless work by our partners at Transparency International Ukraine.

Throughout 2017, Open Ownership worked to create code to convert the XML data from Ukraine’s Unified States Register so it could be included in the OO Register. The completion of this work was announced in January 2018.

As an outcome of this integration work, the Open Ownership team undertook a holistic review of Ukraine’s BO regime. Ukraine’s BO data was offered and regularly updated in the OO Register until September 2020, where certain technical challenges meant new data could not be processed.

Adding Denmark data (August 2018)

BO data from around 300,000 companies in the Central Business Register (Det Centrale Virksomhedsregister, CVR) was added to the Register in August 2018. The CVR contains information on companies from Denmark as well as Greenland.

At that time, Denmark was one of the few European countries that had implemented a public BO register with the CVR, which was created in 2017. Data from the CVR was made accessible for approved users to reuse via an application programming interface (API) using Elasticsearch as well as via the Data Distributor (Datafordeler) platform (see documentation).

Our latest research on Denmark – published in 2023 – lays out how the country’s BOI has been extensively used by businesses and investigative journalists, the financial and real estate sectors, corporate data service providers, and government agencies, and how this was enabled by the comprehensive reforms that Denmark passed to ensure the information is useful, usable, and used.

Enhanced graph visualisation features (July 2019)

After launching the Register with basic visualisation functionality, Laurence Bascle, the then Product Owner, carried out discovery sessions with BO data users which led to a technical sprint that introduced enhanced features.

Powered by Cytoscape’s open-source platform for visualising complex networks, these features gave users the ability to:

• generate graph visualisations of BO networks for people as well as for companies;
• walk “up” the ownership tree via all intermediary corporate ownerships (when available), as well as to walk “down” to show all the companies owned by the target company;
• understand historical data or ceased relationships by greying them out to differentiate them from current ones.

These visualisations offered drag and drop functionality so that users could rearrange the nodes as well as allowing people to zoom in and out to expand or magnify their view.

Although the BO data in the Register was direct ownership information reported to authorities in Denmark, Slovakia, and the UK about the individual or corporate owners one layer up in the chains, the transformation into BODS allowed Open Ownership to link all of this data together to represent full ownership chains in the observable data.

Below is an image showing what the graph visualisations looked like before and after these changes.

Before

After

After announcing these changes, Bascle set out in a later blog post how such visualisation can be a powerful tool for policy makers and implementers, and can display in a “very visual way the pros and cons of different beneficial ownership data collection models”.

Structured bulk data offered (February 2020)

From February 2020, Open Ownership introduced the ability for anyone to download the entire OO Register dataset as a bulk data file.

To start with, over 20 million records were available in the database: every company, person, and ownership or control relationship. Initially, this dataset was formatted in line with version 0.1 of BODS (see example data) and was offered as a large BODS JSON file in the JSONLines format, meaning each BODS statement was a JSON object on a single line of the file.

The download file was regularly republished on a monthly basis, and from August 2023 it was formatted in line with version 0.2 of BODS (see example data).

Thematic guides to how the Register worked (2020)

Starting in July 2020, Open Ownership’s then Technology Lead, Steve Day, documented lessons learned in a series of blog posts about working with BO data. These were intended to share the Open Ownership team’s experiences with data users and publishers in the public and private sectors, and covered the following topics:

• modelling BO data;
• reconciling BO data;
• merging and de-duplicating BO data.

Launch of BODS data analysis tools (March 2022)

To make the most of available BO data, users are often best supported if publishers offer data for download and reuse in a range of formats. In early 2022, Open Ownership already offered the OO Register data in JSON format but knew that some users were not used to working with JSON data.

Harnessing a range of open-source tools and libraries, Open Ownership launched the BODS data analysis tools as an open-source service where users could download open datasets published in line with BODS in a range of formats released using open licences.

Open Ownership still advocates that BO data should be collected, stored, and shared as structured data and standardised in line with a standard such as BODS. Doing so allows the data to be more easily analysed and linked with other datasets, as well supporting its conversion into open formats.

Data from the OO Register and an initial dataset released by Latvia were the first to be made available via the BODS data analysis tools. By flattening and reformatting BODS JSON data, we were able to offer these datasets in the open CSV, SQLite, and PostgreSQL formats as well as creating files to analyse via Google’s Big Query or Datasette.

This work was made possible by Flatterer, an opinionated JSON converter which tries to make a useful relational output for data analysis, created by our technical partners at Open Data Services. Learn more about Flatterer by looking at the launch post, documentation, or code.

New analysis notebook (August 2022)

Introducing Open Ownership’s first data analysis notebook

To show how data from the BODS data analysis tools could be harnessed to understand and swiftly gain insights from data about who owns and controls companies, Open Ownership unveiled our first data analysis notebook.

This free, open-source notebook helps any user get immediate answers to questions relating to the detail, coverage, and timeliness of data published in line with BODS. The structure of this notebook – and of the example feedback report – aligns with the Open Ownership Principles for effective beneficial ownership disclosure.

In order to create the notebook, we developed two main components in the underlying code:

• a Python module, qbods.py, which contains a set of functions for reading, summarising, and analysing BODS data;
• an iPython notebook, latvia_demo.ipynb, which contains code to run a subset of the functions on an initial dataset released by the Register of Enterprises of the Republic of Latvia, with accompanying text.

Following this technical development, our Data Analyst Lewis Spurgin created a sample report on Latvian data with findings and visualisations using the Deepnote collaborative data notebook platform. The analyses presented in this sample report are a subset of a full set of queries, which can be used to analyse any BODS dataset. The notebook code can also be run locally or by using Google Colab.

BODS data analysis tools updated, dashboards launched (December 2022)

As part of the work to redevelop the OO Register backend code (see below), we started republishing data in line with version 0.2 of BODS rather than version 0.1. This allowed us to take advantage of the publicationDetails property added in version 0.2 to split up the OO Register dataset into separate BODS datasets for Denmark, Slovakia, and the UK to support country-based analysis or projects. We also introduced Parquet as an additional download format.

To provide insights into these national sources and to judge the quality of the BO data published, Open Ownership developed a set of live analysis dashboards which could be used to visualise data from Denmark, Slovakia, and the UK.

~

The code for ingesting, mapping, and transforming BO data from Denmark, Slovakia, and the UK in line with version 0.2 of BODS was separated into different GitHub repositories for each country:

• Denmark: ingester / sources / transformer;
• Slovakia: ingester / sources / transformer;
• UK PSC: ingester / sources / transformer.

Challenges with incorporating France data (June 2023)

During the first half of 2023, Open Ownership and our technical partners at Open Data Services worked to understand more about the nature and coverage of France’s national public BO data published by the National Institute of Industrial Property (INPI). Our intention was to map the data to BODS and incorporate it into the Register. Unfortunately, our efforts did not pay off due to issues with accessing the data through the INPI API, as well as data quality issues we discovered within the data itself. Read more about this work and the challenges we faced.

Data use guide published and training sessions held (July 2023)

Publishing good documentation and engaging with users helps drive usage of structured and interoperable BO data. To increase understanding of BODS datasets and Open Ownership’s tools, including the OO Register, we produced a step-by-step guide for BO data users, explaining what each tool was designed to do and how to use them. We also ran a series of three training sessions for users, which were recorded and uploaded to YouTube.

The benefits and advantages of structured and interoperable beneficial ownership data

The Open Ownership Register and tools to work with its data

Visualising and telling stories with beneficial ownership data

New Register backend (August 2023)

After a technical project lasting more than a year, the technology behind the OO Register was updated to be faster, leaner, and more efficient, as well as publishing more detailed and up-to-date BO data. By moving to a new technology stack, a new approach to streaming and fully storing data in BODS format, the OO Register was now able to export data 100 times faster and for a fraction of the previous cost.

This was achieved by leveraging a range of services from Amazon Web Services, such as Kinesis and S3, as well as making use of Elasticsearch to capture both raw data and transformed BODS version 0.2 data. This was thanks to free credits granted to us via Elastic’s philanthropic work (see the code or video recording to learn more).

Ownership technology showcase 5

Legal Entity Identifiers added to Register (September 2023)

Having reliable identifiers present in multiple datasets is the best way to be certain that the same entity is being referred to, and to link together a range of information about the entity and its activities. Since its launch, the OO Register has offered OpenCorporates IDs for companies alongside other national identifiers where their company numbers had already been linked to an OpenCorporates ID.

Thanks to an existing mapping process between OpenCorporates IDs and the Legal Entity Identifier (LEI) from the Global Legal Entity Identifier Foundation, Open Ownership was able to add LEIs for more than 160,000 companies to enrich our BODS datasets, enabling greater connectivity between international BO and corporate ownership datasets.

Nigeria Data Challenge (September 2023)

To encourage data enthusiasts to explore available sources of BO data and create visual representations to explain their findings to the public, Open Ownership launched a Nigeria Data Challenge which solicited a number of exceptional responses. Two submissions describing how public access and data quality influence investigations were later published on the Open Ownership website.

Adding Armenia’s EITI BODS data (December 2023)

First launched in 2020, Armenia’s Multi-Sector Register started capturing BO data in line with version 0.2 of BODS later the following year. This followed significant support from Open Ownership to the Armenian government agencies overseeing the country’s BO reform process during 2019 and 2020. Armenia also made use of the open-source BODS visualisation library to create automatic diagrams of BO networks.

To demonstrate how the collection of standardised BO data in line with BODS helps countries leverage a range of open-source tools, Open Ownership incorporated data disclosed by a number of extractive-industry companies listed by the EITI team in Armenia into the OO Register. This involved creating a new process by which static sources of BODS data could be added to the OO Register in the future.

Likewise, to support Armenia’s innovative use of the alternateNames field in version 0.2 of BODS, Open Ownership also expanded the functionality of the OO Register so that people could find these Armenian companies, as well as millions of other companies, by searching for their alternative names.

Closure of OO Register public website (November 2024)

During 2024, more than seven years since its launch, the Open Ownership team decided to close the OO Register public website, instead choosing to focus our expertise in supporting partners and others to standardise and structure data by accelerating our work on the Beneficial Ownership Data Standard rather than running the OO Register.

From 29 November 2024, users will no longer be able to visit register.openownership.org. The main Open Ownership website, our bulk datasets, and all our other technology products will remain available as normal.

Next page: Data partnerships and experiments