Storage control

Control storage of the system

Common objects

fs_entry: * format: json object * fields: * key: hash: string or null * path: path to pcap on filesystem: string * file_size: size of the pcap file in bytes: 64-bit unsigned * start_ts: epoch (nanoseconds) to first packet in the pcap file: 64-bit unsigned or null

Paging response: * format: json object * fields: * page: current page * page_count: number of pages * total_count: total number of results * data: array of fs_entry object * query arguments * page: selects the page * results_per_page: sets the maximum number of entries displayed per page, default: 5000

API

Get size of metadata partition on the disk (/data partition) * path: /storage/metadata * method: GET * data: json { "capacity_bytes": 2343242, "occupied_bytes:": 3243 } * capacity_bytes - size of the partition * occupied_bytes - how much of the partition is occupied

Display storage information * path: /storage/data * method: GET * data * format: json object * fields: * disk_usage * used: used space in bytes * total: total space in bytes * cleanup_in_progress: true if automatic cleanup is in progress * enable_metadata_cleanup: if set to true, metadata will be cleaned up together with the files as part of the automatic cleanup process; if false, only files are getting removed * age_cleanup: change parameters of automatic files/meta cleanup * mode: files, meta, both, disabled * max_age_hours: files/meta older than max_age_hours are selected for the deletion * constraint: 5 < max_age_hours < 260000 * check_interval_hours: how often age of files/meta is checked against the max_age_hours * constraint: 0 < check_interval_hours < 86400 * example: json { "disk_space": { "total": 17056493568, "used": 15184343040 }, "cleanup_in_progress": false, "enable_automatic_metadata_cleanup": true, "age_cleanup": { "mode": "both", "max_age_hours": 7620, "check_interval_hours": 6 } }

• Change storage configuration
  • path: /storage/data
  • method: PUT
  • data
    • format: json object
    • example:

    {
        "enable_automatic_metadata_cleanup": false,
        "age_cleanup":
        {
            "mode": "both",
            "max_age_hours": 7620,
            "check_interval_hours": 6
        }
    }

• Display captures
  • path: /storage/data/captures
  • method: GET
  • data
    • format: json object
    • fields:
      • page: unsigned integer (1…n), default: 1
      • page_count: unsigned integer, default: 5000
      • order: “asc” (ascending) | “desc” (descending) (default: “asc”)
      • data
        • paging response object
      • example javascript { "page":1, "page_count":1, "total_count":4, "data": [ { "key":"d83e3d630351e1ec752711e92fedcaf977e7dc1b1aa9698f71a4cb41675049ce", "path":"/data/captures/1.pcap", "file_size":1508, "start_ts":1102274184312453000, "end_ts":1102274184312453000, "pending_deletion": false }, { "key":"d83e3d630351e1ec752711e92fedcaf977e7dc1b1aa9698f71a4cb41675049cf", "path":"/data/captures/2.pcap", "file_size":3202, "start_ts":1102274184317113000, "end_ts":1102274184312453000, "pending_deletion": false }, { "key":"d83e3d630351e1ec752711e92fedcaf977e7dc1b1aa9698f71a4cb41675049cg", "path":"/data/captures/3.pcap", "file_size":4442, "start_ts":1102274184227453000, "end_ts":1102274184312453000, "pending_deletion": false }, { "key":null, "path":"/data/captures/3.pcap", "file_size":3339, "start_ts":null, "end_ts":null, "pending_deletion": false } ] }
• Filter by path
  • path: /storage/data/captures
  • method: GET
  • header: paths: [“path-1”, …, “path-n”]
    • if one of the paths don’t exist, it will return 404
  • data
    • paging response object
    • order: by path ascending
• Filter by time
  • path: /storage/data/captures
  • method: GET
  • argument:
    • start_ts: epoch (nanoseconds): 64-bit unsigned
    • end_ts: epoch (nanoseconds): 64-bit unsigned
      • restriction: both start_ts, end_ts must be present
    • order: “asc” (ascending) | “desc” (descending) (default: “asc”)
  • data
    • paging response object
    • order: by start_ts ascending
  • notes
    • entries without timestamp will be skipped
• Import pcap-ng

  • path: /storage/data/captures/imported

  • method: GET

  • Response: ```json { “message”:“Success”, “data”: { “status”:“success”, “message”:“Successfully imported pcap-ng file.” } }

    ```

• Import pcap-ng
  • path: /storage/data/captures
  • method: POST
  • data: pcap-ng file to import
  • behavior
    • start an async procedure of the imported file
    • the status of this procedure is stored and should be retrived from GET /storage/data/captures/imported
• Delete by path
  • path: /storage/data/captures
  • method: DELETE
  • header: paths: [“path-1”, …, “path-n”]
  • argument:
    • list: string flag (true/false): returns the list of deleted captures (default: false)
  • behavior
    • deletion is done asynchronously
    • meanwhile, the field, pending_deletion is set to true
• Delete by time
  • path: /storage/data/captures
  • method: DELETE
  • argument:
    • start_ts: epoch (nanoseconds): 64-bit unsigned
    • end_ts: epoch (nanoseconds): 64-bit unsigned
    • list: string flag (true/false): returns the list of deleted captures (default: false)
  • behavior
    • deletion is done asynchronously
    • meanwhile, the field, pending_deletion is set to true
• Delete all
  • path: /storage/data/captures
  • method: DELETE
  • behavior
    • deletion is done asynchronously
    • meanwhile, the field, pending_deletion is set to true
  • restriction: doesn’t return list of deleted items
• Extract captures
  • path: /storage/data/captures/joined
  • method: GET
  • header: paths: [“path-1”, …, “path-n”]
    • if one of the paths don’t exist, it will return 404
    • if the frontend is unable to send headers, token can be used instead (*): GET /storage/data/captures/joined?token=fcfa8093cf491577d84fa8a37572cc3f
  • arguments
    • start_ts: epoch (nanoseconds): 64-bit unsigned
    • end_ts: epoch (nanoseconds): 64-bit unsigned
    • from: grafana timestamp format
    • to: grafana timestamp format
    • var-esinterface: interface filters; there can be more than one field with this name (TODO: not implemented)
    • var-filters and var-or*: group of filters can be specified by this parameter. Each parameter name will form a group of ANDd values. Then, each group will be ORd as a final expression.
    • var-need_adv_filter: if set to true, then tshark will be used for filtering
    • var-BPF: BPF filter to be applied; zero or one fields of this name can be in the query
    • var-BPF_op: currently not used. BPF expressions will always be ORd to final expression.
    • token: string (*)
    • html_errors: bool (“true” or “false”): returns errors as HTML
  • data: joined captures (pcap)
  • notes:
    • either paths in header or start_ts/end_ts/from/to in arguments must be specified, but not both
    • if paths is not present, either start_ts or end_ts or from/to must be specified, but not both
    • Example for var-filters and var-or*: "var-filters=exp1&var-filters=exp2&var-or=exp3&var-or=exp4&var-or1=exp5" will result in "(exp1 && exp2) || (exp3 && exp4) || (exp5)"
• Get capture extraction token
  • path: /storage/data/captures/joined?get_token=true
  • method: POST
  • header: paths: [“path-1”, …, “path-n”]
  • example:

      {"token": "fcfa8093cf491577d84fa8a37572cc3f"}

Back to Index