Storage control

Control storage of the system

Common objects

fs_entry:

• format: json object
• fields:
  • key: hash: string or null
  • path: path to pcap on filesystem: string
  • file_size: size of the pcap file in bytes: 64-bit unsigned
  • start_ts: epoch (nanoseconds) to first packet in the pcap file: 64-bit unsigned or null
  • end_ts: epoch (nanoseconds) to last packet in the pcap file: 64-bit unsigned or null
  • pending_deletion: true if the file is scheduled for asynchronous deletion: bool

Paging response:

• format: json object
• fields:
  • page: current page
  • page_count: number of pages
  • total_count: total number of results
  • data: array of fs_entry object
• query arguments
  • page: selects the page
  • results_per_page: sets the maximum number of entries displayed per page, default: 5000

API

• Get size of metadata partition on the disk (/data partition)

  • path: /storage/metadata

  • method: GET

  • data:

    {
        "capacity_bytes": 2343242,
        "occupied_bytes": 3243
    }

    • capacity_bytes - size of the partition
    • occupied_bytes - how much of the partition is occupied

• Display storage information

  • path: /storage/data
  • method: GET
  • data
    • format: json object
    • fields:
      • disk_usage
        • used: used space in bytes
        • total: total space in bytes
      • cleanup_in_progress: true if automatic cleanup is in progress
      • enable_metadata_cleanup: if set to true, metadata will be cleaned up together with the files as part of the automatic cleanup process; if false, only files are getting removed
      • age_cleanup: change parameters of automatic files/meta cleanup

        • mode: files, meta, both, disabled

        • max_age_hours: files/meta older than max_age_hours are selected for the deletion

        • constraint: 5 < max_age_hours < 260000

        • check_interval_hours: how often age of files/meta is checked against the max_age_hours

        • constraint: 0 < check_interval_hours < 86400

        • example:

          {
              "disk_space":
              {
                  "total": 17056493568,
                  "used": 15184343040
              },
              "cleanup_in_progress": false,
              "enable_automatic_metadata_cleanup": true,
              "age_cleanup":
              {
                  "mode": "both",
                  "max_age_hours": 7620,
                  "check_interval_hours": 6
              }
          }

• Change storage configuration

  • path: /storage/data
  • method: PUT
  • data

    • format: json object

    • example:

      {
          "enable_automatic_metadata_cleanup": false,
          "age_cleanup":
          {
              "mode": "both",
              "max_age_hours": 7620,
              "check_interval_hours": 6
          }
      }

• Display captures

  • path: /storage/data/captures
  • method: GET
  • data
    • format: json object
    • fields:

      • page: unsigned integer (1…n), default: 1

      • page_count: unsigned integer, default: 5000

      • order: “asc” (ascending) | “desc” (descending) (default: “asc”)

      • data

        • paging response object

      • example

        {
            "page":1,
            "page_count":1,
            "total_count":4,
            "data":
            [
                {
                    "key":"d83e3d630351e1ec752711e92fedcaf977e7dc1b1aa9698f71a4cb41675049ce",
                    "path":"/data/captures/1.pcap",
                    "file_size":1508,
                    "start_ts":1102274184312453000,
                    "end_ts":1102274184312453000,
                    "pending_deletion": false
                },
                {
                    "key":"d83e3d630351e1ec752711e92fedcaf977e7dc1b1aa9698f71a4cb41675049cf",
                    "path":"/data/captures/2.pcap",
                    "file_size":3202,
                    "start_ts":1102274184317113000,
                    "end_ts":1102274184312453000,
                    "pending_deletion": false
                },
                {
                    "key":"d83e3d630351e1ec752711e92fedcaf977e7dc1b1aa9698f71a4cb41675049cg",
                    "path":"/data/captures/3.pcap",
                    "file_size":4442,
                    "start_ts":1102274184227453000,
                    "end_ts":1102274184312453000,
                    "pending_deletion": false
                },
                {
                    "key":null,
                    "path":"/data/captures/3.pcap",
                    "file_size":3339,
                    "start_ts":null,
                    "end_ts":null,
                    "pending_deletion": false
                }
            ]
        }

• Filter by path

  • path: /storage/data/captures
  • method: GET
  • header: paths: [“path-1”, …, “path-n”]
    • if one of the paths don’t exist, it will return 404
  • data
    • paging response object
    • order: by path ascending

• Filter by time

  • path: /storage/data/captures
  • method: GET
  • argument:
    • start_ts: epoch (nanoseconds): 64-bit unsigned
    • end_ts: epoch (nanoseconds): 64-bit unsigned
      • restriction: both start_ts, end_ts must be present
    • order: “asc” (ascending) | “desc” (descending) (default: “asc”)
  • data
    • paging response object
    • order: by start_ts ascending
  • notes
    • entries without timestamp will be skipped

• Get pcap-ng import status

  • path: /storage/data/captures/imported

  • method: GET

  • Response:

    {
        "message":"Success",
        "data":
        {
            "status":"success",
            "message":"Successfully imported pcap-ng file."
        }
    }

• Import pcap-ng

  • path: /storage/data/captures
  • method: POST
  • data: pcap-ng file to import
  • behavior
    • start an async procedure of the imported file
    • the status of this procedure is stored and should be retrived from GET /storage/data/captures/imported

• Delete by path

  • path: /storage/data/captures
  • method: DELETE
  • header: paths: [“path-1”, …, “path-n”]
  • argument:
    • list: string flag (true/false): returns the list of deleted captures (default: false)
  • behavior
    • deletion is done asynchronously
    • meanwhile, the field, pending_deletion is set to true

• Delete by time

  • path: /storage/data/captures
  • method: DELETE
  • argument:
    • start_ts: epoch (nanoseconds): 64-bit unsigned
    • end_ts: epoch (nanoseconds): 64-bit unsigned
    • list: string flag (true/false): returns the list of deleted captures (default: false)
  • behavior
    • deletion is done asynchronously
    • meanwhile, the field, pending_deletion is set to true

• Delete all

  • path: /storage/data/captures
  • method: DELETE
  • behavior
    • deletion is done asynchronously
    • meanwhile, the field, pending_deletion is set to true
  • restriction: doesn’t return list of deleted items

• Extract captures

  • path: /storage/data/captures/joined
  • method: GET
  • header: paths: [“path-1”, …, “path-n”]

    • if one of the paths don’t exist, it will return 404

    • if the frontend is unable to send headers, token can be used instead (*):

         GET /storage/data/captures/joined?token=fcfa8093cf491577d84fa8a37572cc3f

  • arguments
    • start_ts: epoch (nanoseconds): 64-bit unsigned
    • end_ts: epoch (nanoseconds): 64-bit unsigned
    • from: grafana timestamp format
    • to: grafana timestamp format
    • var-filters and var-or*: group of filters can be specified by this parameter. Each parameter name will form a group of ANDd values. Then, each group will be ORd as a final expression.
    • var-need_adv_filter: if set to true, then tshark will be used for filtering
    • var-BPF: BPF filter to be applied; zero or one fields of this name can be in the query. BPF expressions will always be ORd to the final expression.
    • token: string (*)
    • html_errors: bool (“true” or “false”): returns errors as HTML
  • data: joined captures (pcap)
  • notes:
    • either paths in header or start_ts/end_ts/from/to in arguments must be specified, but not both
    • if paths is not present, either start_ts or end_ts or from/to must be specified, but not both
    • Example for var-filters and var-or*: "var-filters=exp1&var-filters=exp2&var-or=exp3&var-or=exp4&var-or1=exp5" will result in "(exp1 && exp2) || (exp3 && exp4) || (exp5)"

• Get capture extraction token

  • path: /storage/data/captures/joined?get_token=true

  • method: POST

  • body:

    [
      "path-1",
      "path-n"
    ]

  • response:

    {"token": "fcfa8093cf491577d84fa8a37572cc3f"}

Back to Index