I Failed the Turing Test

The Graph is a protocol for indexing and querying Blockchain data. Currently, The Graph has the legacy version and the decentralized version. The legacy version is a centralized and managed service hosted by The Graph, and it will eventually be shutdown in the future. The decentralized version (aka The Graph Network) consists of 4 major roles: Developers, Indexers, Curators, and Delegators.

ref:
https://thegraph.com/

Video from Finematics
https://finematics.com/the-graph-explained/

Terminologies

• Subgraph: defines which data should be indexed and how they will be stored and queried.
• Developer: designs and implements subgraphs.
• Indexer: stakes GRT and operates Graph Nodes to index data and serve queries.
  • https://thegraph.com/docs/indexing
  • https://github.com/graphprotocol/graph-node
• Delegator: stakes GRT to indexers to earn rewards, so a delegator doesn't have to run a Graph Node.
  • https://thegraph.com/docs/delegating
• Curator: stakes GRT to signal which subgraphs should be indexed.
  • https://thegraph.com/docs/curating
• Consumer: pays GRT for querying data from indexers.

Define a Subgraph

A subgraph defines one or more entities of indexed data: what kind of data on the blockchain you want to index for faster querying. Once deployed, subgraphs could be queried by dApps to fetch blockchain data to power their frontend interfaces. Basically, a subgraph is like a database, and an entity (a GraphQL type) is like a table in RDBMS.

A subgraph definition consists of 3 files:

• subgraph.yaml: a YAML file that defines the subgraph manifest and metadata.
• schema.graphql: a GraphQL schema that defines what data (entities) are stored, and how to query it via GraphQL.
• mappings.ts: AssemblyScript code that translates blockchain data (events, blocks, or contract calls) to GraphQL entities.

GraphQL Schema

First, we need to design our GraphQL entity schemas (the data model) which mainly depends on how you want to query the data instead of how the data are emitted from the blockchain. GraphQL schemas are defined using the GraphQL schema language.

Here're some notes about subgraph's GraphQL schema:

• For each entity in a subgraph, a id field is required and it must be a string.
• A field with a ! suffix means it's required, otherwise it's optional.
  • https://thegraph.com/docs/developer/create-subgraph-hosted#optional-and-required-fields
• There is a special field type Bytes for storing Ethereum hashes and addresses.
  • https://thegraph.com/docs/developer/create-subgraph-hosted#built-in-scalar-types
• An entity may have relationships to one or more other entities (one-to-one, one-to-many, or many-to-many) which is similar to how you design tables in a relational database.
  • https://thegraph.com/docs/developer/create-subgraph-hosted#entity-relationships
• Subgraphs also support full-text search (since the storage backend is PostgreSQL). Though, it might only has few use cases in the world of blockchains.
  • https://thegraph.com/docs/developer/create-subgraph-hosted#defining-fulltext-search-fields

An example of schema.graphql:

type Market @entity {
  id: ID!
  baseToken: Bytes!
  pool: Bytes!
  feeRatio: BigInt!
  tradingFee: BigDecimal!
  tradingVolume: BigDecimal!
  blockNumberAdded: BigInt!
  timestampAdded: BigInt!
}

type Trader @entity {
  id: ID!
  realizedPnl: BigDecimal!
  fundingPayment: BigDecimal!
  tradingFee: BigDecimal!
  badDebt: BigDecimal!
  totalPnl: BigDecimal!
  positions: [Position!]! @derivedFrom(field: "traderRef")
}

type Position @entity {
  id: ID!
  trader: Bytes!
  baseToken: Bytes!
  positionSize: BigDecimal!
  openNotional: BigDecimal!
  openPrice: BigDecimal!
  realizedPnl: BigDecimal!
  tradingFee: BigDecimal!
  badDebt: BigDecimal!
  totalPnl: BigDecimal!
  traderRef: Trader!
}

ref:
https://thegraph.com/docs/developer/create-subgraph-hosted#the-graphql-schema
https://thegraph.com/docs/developer/graphql-api
https://graphql.org/learn/schema/

It's also worth noting that The Graph supports Time-travel queries. We can query the state of your entities for an arbitrary block in the past:

{
  positions(
    block: {
      number: 1234567
    },
    where: {
      trader: "0x5abfec25f74cd88437631a7731906932776356f9"
    }
  ) {
    id
    trader
    baseToken
    positionSize
    openNotional
    openPrice
    realizedPnl
    fundingPayment
    tradingFee
    badDebt
    totalPnl
    blockNumber
    timestamp
  }
}

ref:
https://thegraph.com/docs/developer/graphql-api#time-travel-queries

Subgraph Manifest

Second, we must provide a manifest to tell The Graph which contracts we would like to listen to, which contract events we want to index. Also a mapping file that instructs The Graph on how to transform blockchain data into GraphQL entities.

A template file of subgraph.yaml:

specVersion: 0.0.2
description: Test Subgraph
repository: https://github.com/vinta/my-subgraph
schema:
  file: ./schema.graphql
dataSources:
  - kind: ethereum/contract
    name: ClearingHouse
    network: {{ network }}
    source:
      abi: ClearingHouse
      address: {{ clearingHouse.address }}
      startBlock: {{ clearingHouse.startBlock }}
    mapping:
      kind: ethereum/events
      apiVersion: 0.0.4
      language: wasm/assemblyscript
      file: ./src/mappings/clearingHouse.ts
      entities:
        - Protocol
        - Market
        - Trader
        - Position
      abis:
        - name: ClearingHouse
          file: ./abis/ClearingHouse.json
      eventHandlers:
        - event: PoolAdded(indexed address,indexed uint24,indexed address)
          handler: handlePoolAdded
        - event: PositionChanged(indexed address,indexed address,int256,int256,uint256,int256,uint256)
          handler: handlePositionChanged

Since we would usually deploy our contracts to multiple chains (at least one for mainnet and one for testnet), so we could use a template engine (like mustache.js) to facilate deployment.

$ cat configs/arbitrum-rinkeby.json
{
    "network": "arbitrum-rinkeby",
    "clearingHouse": {
        "address": "0xYourContractAddress",
        "startBlock": 1234567
    }
}

# generate the subgraph manifest for different networks
$ mustache configs/arbitrum-rinkeby.json subgraph.template.yaml > subgraph.yaml
$ mustache configs/arbitrum-one.json subgraph.template.yaml > subgraph.yaml

It's worth noting that The Graph Legacy (the Hosted Service) supports most of common networks, for instance, mainnet, rinkeby, bsc, matic, arbitrum-one, and optimism. However, The Graph Network (the decentralized version) only supports Ethereum mainnet and rinkeby.

You could find the full list of supported networks on the document:
https://thegraph.com/docs/developer/create-subgraph-hosted#from-an-existing-contract

Mappings

Mappings are written in AssemblyScript and will be compiled to WebAssembly (WASM) when deploying. AssemblyScript's syntax is similar to TypeScript, but it's actually a completely different language.

For each event handler defined in subgraph.yaml under mapping.eventHandlers, we must create an exported function of the same name. What we do in a event handler is basically:

1. Creating new entities or loading existed ones by id.
2. Updating fields of entities from a blockchain event.
3. Saving entities to The Graph.
   • It's not necessary to load an entity before updating it. It's fine to simply create the entity, set properties, then save. If the entity already exists, changes will be merged automatically.

export function handlePoolAdded(event: PoolAdded): void {
    // upsert Protocol
    const protocol = getOrCreateProtocol()
    protocol.publicMarketCount = protocol.publicMarketCount.plus(BI_ONE)

    // upsert Market
    const market = getOrCreateMarket(event.params.baseToken)
    market.baseToken = event.params.baseToken
    market.pool = event.params.pool
    market.feeRatio = BigInt.fromI32(event.params.feeRatio)
    market.blockNumberAdded = event.block.number
    market.timestampAdded = event.block.timestamp

    // commit changes
    protocol.save()
    market.save()
}

export function handlePositionChanged(event: Swapped): void {
    // upsert Market
    const market = getOrCreateMarket(event.params.baseToken)
    market.tradingFee = market.tradingFee.plus(swappedEvent.fee)
    market.tradingVolume = market.tradingVolume.plus(abs(swappedEvent.exchangedPositionNotional))
    ...

    // upsert Trader
    const trader = getOrCreateTrader(event.params.trader)
    trader.tradingFee = trader.tradingFee.plus(swappedEvent.fee)
    trader.realizedPnl = trader.realizedPnl.plus(swappedEvent.realizedPnl)
    ...

    // upsert Position
    const position = getOrCreatePosition(event.params.trader, event.params.baseToken)
    const side = swappedEvent.exchangedPositionSize.ge(BD_ZERO) ? Side.BUY : Side.SELL
    ...

    // commit changes
    market.save()
    trader.save()
    position.save()
}

We can also access contract states and call contract functions at the current block (even ). Though the functionality of calling contract functions is limited by @graphprotocol/graph-ts, it's not as powerful as libraries like ethers.js. And no, we cannot import ethers.js in mappings, as mappings are written in AssemblyScript. However, contract calls are quite "expensive" in terms of indexing performance. In extreme cases, some indexers might avoid syncing a very slow subgraph, or charge a premium for serving queries.

export function handlePoolAdded(event: PoolAdded): void {
    ...
    const pool = UniswapV3Pool.bind(event.params.pool)
    market.poolTickSpacing = pool.tickSpacing()
    ...
}

ref:
https://thegraph.com/docs/developer/create-subgraph-hosted#writing-mappings
https://thegraph.com/docs/developer/assemblyscript-api

In addition to event handlers, we're also able to define call handlers and block handlers. A call handler listens to a specific contract function call, and receives input and output of the call as the handler argument. On the contrary, a block handler will be called after every block or after blocks that match a predefined filter - for every block which contains a call to the contract listed in dataSources.

ref:
https://thegraph.com/docs/developer/create-subgraph-hosted#defining-a-call-handler
https://thegraph.com/docs/developer/create-subgraph-hosted#block-handlers

Here're references to how other projects organize their subgraphs:
https://github.com/Uniswap/uniswap-v3-subgraph
https://github.com/Synthetixio/synthetix-subgraph
https://github.com/mcdexio/mai3-perpetual-graph

Deploy a Subgraph

Deploy to Legacy Explorer

Before deploying your subgraph to the Legacy Explorer (the centralized and hosted version of The Graph), we need to create it on the Legacy Explorer dashboard.

Then run the following commands to deploy:

$ mustache configs/arbitrum-rinkeby.json subgraph.template.yaml > subgraph.yaml

$ graph auth --product hosted-service <YOUR_THE_GRAPH_ACCESS_TOKEN>
$ graph deploy --product hosted-service <YOUR_GITHUB_USERNAME>/<YOUR_SUBGRAPH_REPO>

ref:
https://thegraph.com/docs/developer/deploy-subgraph-hosted

Deploy to Subgraph Studio

When we deploy a subgraph to Subgraph Studio (the decentralized version of The Graph), we just push it to the Studio where we're able to test it. Versus, when we "publish" a subgraph in Subgraph Studio, we are publishing it on-chain. Unfortunately, Subgraph Studio only supports Ethereum Mainnet and Rinkeby testnet currently.

$ graph auth --studio <YOUR_SUBGRAPH_DEPLOY_KEY>
$ graph deploy --studio <YOUR_SUBGRAPH_SLUG>

ref:
https://thegraph.com/docs/developer/deploy-subgraph-studio

Token Economics

Before we talk about the token economics of GRT token, it's important to know that the following description only applies to The Graph Network, the decentralized version of The Graph. Also, the name of The Graph Network is a bit ambiguous, it is not a new network or a new blockchain, instead, it is a web service that charges HTTP/WebSocket API calls in GRT token.

To make GRT token somehow valuable, when you query data (through GraphQL APIs) from The Graph Network, you need to pay for each query in GRT. First, you have to connect your wallet and create an account on Subgraph Studio to obtain an API key, then you deposit some GRT tokens into the account's billing balance on Polygon since their billing contract is built on Polygon. At the end of each week, if you used your API keys to query data, you will receive an invoice based on the query fees you have generated during this period. This invoice will be paid using GRT available in your balance.

ref:
https://thegraph.com/docs/studio/billing

When it comes to token economics:

• Indexers earn query fees and indexing rewards. GRT would be slashed if indexers are malicious or serve incorrect data. Though, there's no documentation about how exactly slashing works.
• Delegators earn a portion of query fees and indexing rewards by delegating GRT to existing indexers.
• Curators earn a portion of query fees for the subgraphs they signal on by depositing GRT into a bonding curve of a specific subgraph.

ref:
https://thegraph.com/blog/the-graph-grt-token-economics
https://thegraph.com/blog/the-graph-network-in-depth-part-1
https://thegraph.com/blog/the-graph-network-in-depth-part-2

Query Fee

The price of queries will be set by indexers and vary based on cost to index the subgraph, the demand for queries, the amount of curation signal and the market rate for blockchain queries. Though querying data from the hosted version of The Graph is free now.

The Graph has developed a Cost Model (Agora) for pricing queries, and there is also a microtransaction system (Scalar) that uses state channels to aggregate and compress transactions before being finalized on-chain.

ref:
https://github.com/graphprotocol/agora
https://thegraph.com/blog/scalar