Database

TIP

You can find the source code of this package at packages/core-database.

Installation

yarn add @arkecosystem/core-database

Alias

database

Interface

core-database

Implementation

core-database-postgres

Notable Dependencies

Summary

The database package is the primary interaction layer between the raw blockchain data and the various other packages who need to access that layer.

The current database implementation is built upon PostgresQL, a high-performance relational database that facilitates the SQL read queries necessary to ensure the full range of API and webhook functionality.

Usage

Resolving core-database into your plugins after core-database-postgres has loaded in your node will give you an object with several methods designed as top-level entry points to the Postgres database. Keep in mind that the database is largely an internal package, and that the database state alone does not represent the total state of the blockchain. Thus, for all access-related queries it is generally recommended to use the Public API instead. Nonetheless, for bridgechain developers and core developers, the core-database-postgres package offers several methods to access and update your Ark Core database:

  • getBlock(id) accepts a block ID and returns a Block model from crypto with all relevant transactions attached as an array of Transaction models.
  • getLastBlock() returns the last block stored in the database. Keep in mind that this block might not always be the most current block: if a block has just propagated elsewhere, there is a slight delay that block's creation and its inclusion in all nodes across the network. Thus, for accessing the last block, it is often better to use the core-blockchain API, which has better vision into the state of the network.
  • getBlocks(offset, limit) accepts a limit and an offset from maximum height, and returns an array of blocks with related transactions.
  • getBlockHeaders(offset, limit) does the same as getBlocks, but returns exclusively headers. Useful in chain validation.
  • getTransaction(id) takes a transaction ID and returns a Transaction model from crypto.
  • getActiveDelegates(height, delegates) takes as parameters the total list of delegates and the height at which a delegate list should be built. It returns an array of all actively forging delegates, sorted by vote count descending.
  • buildWallets() loads and returns wallets using simple payment verification.
  • loadWallets() loads and returns wallets using database queries.
  • saveWallets(force) saves wallets from memory into the database, along with any changes. If force evaluates to true, the database is cleared of all wallets first — otherwise, the wallets database is kept intact and wallets in the database are updated by wallets in memory if applicable. Keep in mind that calling this outside of very specific context will have no effect, as database changes are detected and rolled back when an Ark Core node falls out of sync with its peers.
  • verifyBlockchain() checks the Postgres database to ensure the following facts:
    • A last block (ie. the block at maximum chain height) is accessible
    • Last block height is equal to the block count
    • Number of stored transactions equals the sum of all block transaction counts
    • Sum of all transaction fees equals all block fees
    • Sum of all transaction amounts equals all block total amounts

Behind the Scenes

At the top level of the package, an instance of PostgresConnection is created from the neighboring connection.js file and loaded into the database manager found in the interface.

The connection from Ark Core to the Postgres database is created though the connection's make method:

async make() {
    if (this.db) {
      throw new Error('Database connection already initialised')
    }

    logger.debug('Connecting to database')

    this.queuedQueries = null
    this.cache = new Map()

    try {
      await this.connect()
      await this.__registerQueryExecutor()
      await this.__runMigrations()
      await this.__registerModels()
      await super._registerRepositories()
      await super._registerWalletManager()

      this.blocksInCurrentRound = await this.__getBlocksForRound()

      return this
    } catch (error) {
      app.forceExit('Unable to connect to the database!', error)
    }
  }

As the code above demonstrates, the PostgresQL database connection consists of the following major parts:

  • QueryExecutor: this class is responsible for executing queries on the databases. Its various methods correspond to how many results the query should expect as its return value: none, one, oneOrNone, many, manyOrNone, and any.
  • Migrations: this JavaScript file loads the various migration SQL files necessary for the PostgresQL DB and executes them if necessary. In general, the concept of migrations is used across software programming to refer to the process of creating and updating how data is saved in a database. In other words, as applications evolve and their data needs change, data representations migrate from one form to another, with each such migration represented in one SQL query.
  • Models: these JavaScript classes guide the serialization process as raw PostgresQL query results are transformed into data objects for use elsewhere in Ark Core. Keep in mind that, unlike the data models available in Core's crypto library, these data models contain strictly data, not methods. The sole responsibility of the models in core-database-postgres is to ensure that the raw data results returned from Postgres queries can be accessed in JavaScript without incident.
  • Repositories: These repositories combine Postgres queries with database models to produce JavaScript object responses to data queries throughout Core.
  • WalletManager: an in-memory access point for wallet information. Changes in wallet balances are recorded in the WalletManager first, then saved into the database in batches to improve performance.

Interface Methods

These are the functions outlined in the database interface:

  • verifyBlockchain
  • getActiveDelegates
  • buildWallets
  • saveWallets
  • saveBlock
  • enqueueSaveBlock
  • enqueueDeleteBlock
  • enqueueDeleteRound
  • commitQueuedQueries
  • deleteBlock
  • getBlock
  • getLastBlock
  • getBlocks
  • getTopBlocks
  • getRecentBlockIds
  • saveRound
  • deleteRound

Rewriting the core-database layer should be done with caution. Although the Postgres database implementation might require more computational resources than a lower-level datastore, this complexity brings with it a streamlining of developer experience through the use of well-known SQL queries. This is particularly important given the many ways in which Ark Core data can be accessed by external applications — from the Public API to webhooks to GraphQL.

Postgres Database Defaults

{
  initialization: {
    capSQL: true,
    promiseLib: require('bluebird'),
    noLocking: process.env.NODE_ENV === 'test',
  },
  connection: {
    host: process.env.ARK_DB_HOST || 'localhost',
    port: process.env.ARK_DB_PORT || 5432,
    database:
      process.env.ARK_DB_DATABASE || `ark_${process.env.ARK_NETWORK_NAME}`,
    user: process.env.ARK_DB_USERNAME || 'ark',
    password: process.env.ARK_DB_PASSWORD || 'password',
  },
}

Database Interface Defaults

{
  snapshots: `${process.env.ARK_PATH_DATA}/snapshots/${
    process.env.ARK_NETWORK_NAME
  }`,
}
Last Updated: 12/7/2018, 1:53:36 AM