If you are looking for a Moodle function to do this then you can call:

$cat = core_course_category::get($categoryid);
$courseids = $cat->get_courses(['recursive', 'idonly']);
$userids = [];
foreach ($courseids as $courseid) {
    $context = context_course::instance($courseid);
    $courseusers = get_enrolled_users($context, '', 0, 'u.id');
    $userids = array_merge($userids, array_keys($courseusers));
}

However, this is very inefficient - you'd be better off writing a custom SQL query that, based on a list of course IDs, will generate a list of registered users for all those courses (look inside the code) get_enrolled_users() to see how it's constructed Such SQL query).