If you need to suck a lot of data out of Zoho CRM, there’s only one option you should consider: the bulk read API. If you try and use the get records API, you’ll quickly use up all your API calls. Let’s do some quick maths to compare the options. Assuming you have 500k records in your CRM, if you try and retrieve the data using getRecords, you’ll use 500k / 200 (maximum page size) = 2500 API calls. If you use the bulk read API, you will use a grand total of 150 API calls. Massive difference.
When you log into the Smooth Messenger app for the first time, we download all your leads, contacts, deals and SMS records so that you have their phone numbers when you go to send an SMS/make a call. If we tried to use getRecords, you’d get pretty annoyed with us because that could easily wipe out your entire API call quota for the day. Because we use the bulk read API, it’s not a problem at all.
How the bulk read API works
Unlike getRecords which is a synchronous API, the bulk read API gets back to you in a slightly delayed fashion. You submit an initial API request kicking off a bulk read job. In that request payload, you specify a URL where the results of the job should be sent. A little while later, Zoho’s servers deliver a zipped CSV file to that endpoint. Your code can unpackage the CSV goodness and quickly populate your database.
Even though it’s async, it’s way faster than using getRecords because you get so much data at once (up to 200k records per request).
What the code looks like
Here’s an example of how we use bulk read. Take note of the Modified_Time filter which makes sure we only sync data that has changed since the last sync.
public static function sendBulkFetchRequest($userId, $module, $page = 1, $token = null, $modifiedHeader = false)
{
$user = User::find($userId);
$accessToken = $token;
if (!$token) {
$accessToken = self::generateOAuthAccessToken($userId);
}
$apiDomain = self::getUserMetaData($userId, 'api_domain');
$url = $apiDomain . "/crm/bulk/v2/read";
$headers = array(
"Authorization: Zoho-oauthtoken $accessToken",
"Content-Type: application/json",
);
$query_params = array(
'module' => $module,
'page' => $page
);
$modifiedAfter = null;
if ($modifiedHeader) {
$lastSync = self::getOrgLastSync($user->zoho_org_id);
if ($lastSync) {
$modifiedAfter = $lastSync;
}
}
if ($modifiedAfter) {
$query_params['criteria'] = array (
'api_name' => 'Modified_Time',
'value' => $modifiedAfter,
'comparator' => 'greater_equal'
);
}
$callBackUrl = url('/api/some-endpoint-we-use-to-receive-bulk-requests-not-including-here-because-you-might-attack-it?user_id=' . $userId . '&module=' . $module);
$body = array(
'callback' => array(
'url' => $callBackUrl,
'method' => 'post',
),
'query' => $query_params
);
$body = json_encode($body);
$result = self::curl($url, $body, 'get', $headers);
$bulkJob = json_decode($result, true);
if (isset($bulkJob['data'][0]['details']['id'])) {
$bulkSyncJob = new BulkSyncJob();
$bulkSyncJob->crm_organisation_id = $user->zoho_org_id;
$bulkSyncJob->zoho_job_id = $bulkJob['data'][0]['details']['id'];
$bulkSyncJob->status = 0;
$bulkSyncJob->module = $module;
$bulkSyncJob->save();
}
return true;
}
public static function processBulkDataFromCRM($userId, $jobId, $module)
{
$user = User::where('id', $userId)->first();
$setting = Setting::where('crm_organisation_id', $user->zoho_org_id)->first();
$accessToken = self::generateOAuthAccessToken($userId);
if ($accessToken) {
$apiDomain = self::getUserMetaData($user->id, 'api_domain');
$url = $apiDomain . "/crm/bulk/v2/read/$jobId/result";
$headers = array(
"Authorization: Zoho-oauthtoken $accessToken",
"Content-Type: application/json",
);
$fileName = $userId . '_result.zip';
if (!file_exists(storage_path()."/csvFolder")) {
mkdir(storage_path() . "/csvFolder", 0750, true);
}
$fileName = storage_path() . "/csvFolder/$fileName";
$body = '';
$result = self::curl($url, $body, 'get', $headers);
$downloadedFileCont = $result;
$save = file_put_contents($fileName, $downloadedFileCont);
if ($save) {
$extractFolder = storage_path() . "/csvFolder/extractFolder";
$zip = new \ZipArchive;
$res = $zip->open($fileName);
if ($res === true) {
$zip->extractTo($extractFolder);
$zip->close();
$csvFileRow = 1;
if (($handle = fopen($extractFolder . "/" . $jobId . '.csv', "r")) !== false) {
$csvFilePath = $extractFolder . "/" . $jobId . '.csv';
$fieldNames = [];
if ($module == 'twiliosmsextension0__Sent_SMS') {
while (($data = fgetcsv($handle)) !== false) {
$columnCount = count($data);
$messageFields = [
'Id', 'Sid', 'Message_Date','To', 'From', 'Message', 'Message_Type', 'Owner',
'twiliosmsextension0__Status', 'twiliosmsextension0__MMS_content',
'twiliosmsextension0__Message_Source',
];
$messageFields = MessageCanBeLinkedToModule::getFieldNamesWhereSMSCanBeLinkedTo($setting, $messageFields, 'bulk_sync');
if ($csvFileRow == 1) {
for ($numberOfColumn = 0; $numberOfColumn < $columnCount; $numberOfColumn++) {
if (in_array($data[$numberOfColumn], $messageFields)) {
$fieldNames[] = array('name' => $data[$numberOfColumn], 'index' => $numberOfColumn);
}
}
} else if ($csvFileRow !== 1) {
$msgData = array();
for ($numberOfColumn = 0; $numberOfColumn < $columnCount; $numberOfColumn++) {
foreach ($fieldNames as $fieldName) {
if (in_array($fieldName['name'], $messageFields) && $fieldName['index'] == $numberOfColumn) {
$msgData[$fieldName['name']] = $data[$numberOfColumn];
}
}
}
$msgData = MessageCanBeLinkedToModule::setModuleFieldToSMSRecord($setting, $msgData);
$msgData['twiliosmsextension0__Message_Source'] = isset($msgData['twiliosmsextension0__Message_Source']) ? "bulk-sync ".$msgData['twiliosmsextension0__Message_Source'] : 'bulk-sync';
self::saveMessageFromCRM($userId, $msgData);
}
$csvFileRow++;
}
}
$syncModules = ['Contacts', 'Leads', 'Deals', 'Accounts'];
if (in_array($module ,$syncModules)) {
$phoneFields = self::getModuleWisePhoneFields($userId, $module);
$contactFields = [
'twiliosmsextension0__Number_Field_to_Use_for_Twilio_Messages',
'Id', 'Owner', 'First_Name', 'Last_Name','Email','Account_Name',
'Company'
];
if ($module == 'Deals') {
$contactFields[] = 'Deal_Name';
}
if (count($phoneFields) > 0) {
$contactFields = array_merge($contactFields, $phoneFields);
}
while (($data = fgetcsv($handle)) !== false) {
$columnCount = count($data);
if ($csvFileRow == 1) {
for ($numberOfColumn = 0; $numberOfColumn < $columnCount; $numberOfColumn++) {
if (in_array($data[$numberOfColumn], $contactFields)) {
$fieldNames[] = array('name' => $data[$numberOfColumn], 'index' => $numberOfColumn);
}
}
} else if ($csvFileRow !== 1) {
$contactData = array();
for ($numberOfColumn = 0; $numberOfColumn < $columnCount; $numberOfColumn++) {
foreach ($fieldNames as $fieldName) {
if (in_array($fieldName['name'], $contactFields) && $fieldName['index'] == $numberOfColumn) {
$contactData[$fieldName['name']] = $data[$numberOfColumn];
}
}
}
if ($module == 'Deals') {
$contactData['First_Name'] = isset($contactData['Deal_Name']) ? $contactData['Deal_Name'] : null;
}
if ($module == 'Accounts') {
$contact_data['First_Name'] = isset($contactData['Account_Name']) ? $contactData['Account_Name'] : null;
} else {
$contact_data['Account_Name'] = isset($contactData['Account_Name']) ? $contactData['Account_Name'] : null;
}
$contactData['Company'] = isset($contactData['Company']) ? $contactData['Company'] : null;
$contactData['zoho_org_id'] = $user->zoho_org_id;
$contactData['crm_owner_id'] = isset($contactData['Owner']) && !empty($contactData['Owner']) ? $contactData['Owner'] : '';
self::saveContactLeadFromCRM($userId, $module, $contactData, $phoneFields);
}
$csvFileRow++;
}
}
if (!empty($module) && !in_array($module ,$syncModules) && $module != 'twiliosmsextension0__Sent_SMS') {
$phoneFields = self::getModuleWisePhoneFields($userId, $module);
$contactFields = ['Id', 'Owner','Email','Account_Name','Company'];
$nameFields = ZohoCRM::getModuleWiseNameFields($userId, $module);
$contactFields[] = array_merge($contactFields, $nameFields);
if (count($phoneFields) > 0) {
$contactFields = array_merge($contactFields, $phoneFields);
}
while (($data = fgetcsv($handle)) !== false) {
$columnCount = count($data);
if ($csvFileRow == 1) {
for ($numberOfColumn = 0; $numberOfColumn < $columnCount; $numberOfColumn++) {
if (in_array($data[$numberOfColumn], $contactFields)) {
$fieldNames[] = array('name' => $data[$numberOfColumn], 'index' => $numberOfColumn);
}
}
} else if ($csvFileRow !== 1) {
$contactData = array();
for ($numberOfColumn = 0; $numberOfColumn < $columnCount; $numberOfColumn++) {
foreach ($fieldNames as $fieldName) {
if (in_array($fieldName['name'], $contactFields) && $fieldName['index'] == $numberOfColumn) {
$contactData[$fieldName['name']] = $data[$numberOfColumn];
}
}
}
if ($nameFields) {
foreach($nameFields as $key) {
if (array_key_exists($key, $contactData)) {
$matchingNameFields[$key] = $contactData[$key];
}
}
$name = current(array_filter($matchingNameFields));
$contactData['First_Name'] = isset($name) ? $name : null;
}
$contactData['Company'] = isset($contactData['Company']) ? $contactData['Company'] : null;
if ($module == 'Accounts') {
$contactData['First_Name'] = isset($contactData['Account_Name']) ? $contactData['Account_Name'] : null;
} else {
$contactData['Account_Name'] = isset($contactData['Account_Name']) && !empty($contactData['Account_Name']) ? $contactData['Account_Name']['id'] : null;
}
$contactData['zoho_org_id'] = $user->zoho_org_id;
$contactData['crm_owner_id'] = isset($contactData['Owner']) && !empty($contactData['Owner']) ? $contactData['Owner'] : '';
self::saveContactLeadFromCRM($userId, $module, $contactData, $phoneFields);
}
$csvFileRow++;
}
}
fclose($handle);
if (file_exists($csvFilePath)) {
unlink($csvFilePath);
}
if (file_exists($fileName)) {
unlink($fileName);
}
}
}
}
}
return true;
}