Subject: Help with Streaming and Chunk Processing for Large JSON Data (60 GB)
from Kenna API
Dear Python Experts,
I am working with the Kenna Application's API to retrieve vulnerability
data. The API endpoint provides a single, massive JSON file in gzip format, approximately 60 GB in size. Handling such a large dataset in one go is
proving to be quite challenging, especially in terms of memory management.
I am looking for guidance on how to efficiently stream this data and
process it in chunks using Python. Specifically, I am wondering if there=E2= =80=99s
a way to use the requests library or any other libraries that would allow
us to pull data from the API endpoint in a memory-efficient manner.
Here are the relevant API endpoints from Kenna:
- Kenna API Documentation
<
https://apidocs.kennasecurity.com/reference/welcome>
- Kenna Vulnerabilities Export
<
https://apidocs.kennasecurity.com/reference/retrieve-data-export>
If anyone has experience with similar use cases or can offer any advice, it would be greatly appreciated.
Thank you in advance for your help!
Best regards
Asif Ali
--- MBSE BBS v1.0.8.4 (Linux-x86_64)
* Origin: ---:- FTN<->UseNet Gate -:--- (3:633/280.2@fidonet)