Add Ipfs-DagSize and/or Ipfs-DataSize to gateway responses

[lidel]

This is a placeholder issue for writing spec and tracking implementation.

The need is to expose equivalent of ipfs files stat's Size and CumulativeSize on HTTP gateway somehow, allowing light clients to understand how big is the DAG before fetching it as a CAR etc.

• Ipfs-DagSize total size of the DAG (user data + DAG metadata, IPLD envelopes etc)
• Ipfs-DataSize total size of the raw data (without DAG metadata, IPLD envelopes etc)

We don't want expensive equivalent of ipfs dag stat for Unixfs, we want to leverage hints from UnixFS. It could be limited to formats like UnixFS, or single-block dag-cbor etc, which include total sizes in the root blocks, removing the need for expensive ipfs dag stat of entire DAG.

Implementation options

• 🟢 HTTP Headers
  • Ipfs-DagSize and Ipfs-DataSize headers with value in bytes
  • pro: simple and easy to implement, JS apps already use Content-Type for this (because without compression it matches payload size), we should not break them without giving a replacement header
  • 💡 caveat: unnecessary work on every request, unless we limit them to HEAD responses
• ⛔ Custom content type that returns useful metadata
  • would be opt-in Accept HTTP header or format URL arg)
  • pro: could be leveraged for things other than size info
  • con: complexity, need to decide on JSON schema, harder for gateways and clients to adopt

cc @vasco-santos I know you needed something like this a while ago