Investigation of a Workbench U... Note

Investigation of a Workbench UI Latency Issue

A user reported that their JupyterLab UI on their Workbench becomes slow and unresponsive when running certain notebooks. To quantify the slowness, we held down a key for 15 seconds while running the user's notebook and observed latencies ranging from 1 to 10 seconds, averaging 7.4 seconds. We ruled out CPU oversubscription and network issues as potential causes. Using py-spy, we found that a lot of CPU time was spent on a function called __parse_smaps_rollup in the jupyter-lab process. This function is part of the jupyter_resource_usage extension, which is used to get resource usage information. The function is triggered by an API endpoint called periodically from the UI. The code gets all children processes of the jupyter-lab process recursively, including both the ipykernel Notebook process and all processes created by the Notebook. This function's cost is linear to the number of all children processes. In the reproduction code, we create 96 processes, which is more than the 64 CPUs allocated to the container. This discrepancy causes the parent process to be slow.