Upload large files to SharePoint online document library using PowerShell csom

In this post, we will discuss how to upload large files to SharePoint online document library using PowerShell CSOM. I have followed the PowerShell script from Asad Refai.

Upload large files to SharePoint online document library using PowerShell

Here we have a SharePoint online document library and will upload documents from the local drive. If the file size is less than 9 MB then it will upload directly to the document library. But if it is more than 9MB then it will upload in a chunk.

Below is the full PowerShell script to upload large files to SharePoint online document library. You can write the below PowerShell script by using visual studio code or PowerShell ISE.

$SiteURL = "https://onlysharepoint2013.sharepoint.com/sites/Bhawana/"
$libraryName="MyTestDocLib"
$SPOnlineUserName="Bijay@*********.onmicrosoft.com"
$SPOnlinePassword="***********"
$fileDirectory="C:\Users\Bijaya.Sahoo\Desktop\Bijay\Files"
Try{
Add-Type -Path 'C:\Users\Bijaya.Sahoo\Desktop\Microsoft.SharePoint.Client.dll'
Add-Type -Path 'C:\Users\Bijaya.Sahoo\Desktop\Microsoft.SharePoint.Client.Runtime.dll'
}
catch {
Write-host "No dlls found"
}
Function UploadFileInSlice ($ctx, $libraryName, $fileName, $fileChunkSizeInMB) {
$fileChunkSizeInMB = 9
# Each sliced upload requires a unique ID.
$UploadId = [GUID]::NewGuid()
# Get the name of the file.
$UniqueFileName = [System.IO.Path]::GetFileName($fileName)
# Get the folder to upload into.
$Docs = $ctx.Web.Lists.GetByTitle($libraryName)
$ctx.Load($Docs)
$ctx.Load($Docs.RootFolder)
$ctx.ExecuteQuery()
# Get the information about the folder that will hold the file.
$ServerRelativeUrlOfRootFolder = $Docs.RootFolder.ServerRelativeUrl
# File object.
[Microsoft.SharePoint.Client.File] $upload
# Calculate block size in bytes.
$BlockSize = $fileChunkSizeInMB * 1024 * 1024
# Get the size of the file.
$FileSize = (Get-Item $fileName).length
if ($FileSize -le $BlockSize)
{
# Use regular approach.
$FileStream = New-Object IO.FileStream($fileName,[System.IO.FileMode]::Open, [System.IO.FileAccess]::Read)
$FileCreationInfo = New-Object Microsoft.SharePoint.Client.FileCreationInformation
$FileCreationInfo.Overwrite = $true
$FileCreationInfo.ContentStream = $FileStream
$FileCreationInfo.URL = $UniqueFileName
$Upload = $Docs.RootFolder.Files.Add($FileCreationInfo)
$ctx.Load($Upload)
$ctx.ExecuteQuery()
return $Upload
}
else
{
# Use large file upload approach.
$BytesUploaded = $null
$Fs = $null
Try {
$Fs = [System.IO.File]::Open($fileName, [System.IO.FileMode]::Open, [System.IO.FileAccess]::Read)
$br = New-Object System.IO.BinaryReader($Fs)
$buffer = New-Object System.Byte[]($BlockSize)
$lastBuffer = $null
$fileoffset = 0
$totalBytesRead = 0
$bytesRead
$first = $true
$last = $false
# Read data from file system in blocks.
while(($bytesRead = $br.Read($buffer, 0, $buffer.Length)) -gt 0) {
$totalBytesRead = $totalBytesRead + $bytesRead
# You've reached the end of the file.
if($totalBytesRead -eq $FileSize) {
$last = $true
# Copy to a new buffer that has the correct size.
$lastBuffer = New-Object System.Byte[]($bytesRead)
[array]::Copy($buffer, 0, $lastBuffer, 0, $bytesRead)
}
If($first)
{
$ContentStream = New-Object System.IO.MemoryStream
# Add an empty file.
$fileInfo = New-Object Microsoft.SharePoint.Client.FileCreationInformation
$fileInfo.ContentStream = $ContentStream
$fileInfo.Url = $UniqueFileName
$fileInfo.Overwrite = $true
$Upload = $Docs.RootFolder.Files.Add($fileInfo)
$ctx.Load($Upload)
# Start upload by uploading the first slice.
$s = [System.IO.MemoryStream]::new($buffer)
# Call the start upload method on the first slice.
$BytesUploaded = $Upload.StartUpload($UploadId, $s)
$ctx.ExecuteQuery()
# fileoffset is the pointer where the next slice will be added.
$fileoffset = $BytesUploaded.Value
# You can only start the upload once.
$first = $false
}
Else
{
# Get a reference to your file.
$Upload = $ctx.Web.GetFileByServerRelativeUrl($Docs.RootFolder.ServerRelativeUrl + [System.IO.Path]::AltDirectorySeparatorChar + $UniqueFileName);
If($last) {
# Is this the last slice of data?
$s = [System.IO.MemoryStream]::new($lastBuffer)
# End sliced upload by calling FinishUpload.
$Upload = $Upload.FinishUpload($UploadId, $fileoffset, $s)
$ctx.ExecuteQuery()
Write-Host "File upload complete"
# Return the file object for the uploaded file.
return $Upload
}
else {
$s = [System.IO.MemoryStream]::new($buffer)
# Continue sliced upload.
$BytesUploaded = $Upload.ContinueUpload($UploadId, $fileoffset, $s)
$ctx.ExecuteQuery()
# Update fileoffset for the next slice.
$fileoffset = $BytesUploaded.Value
}
}
} #// while ((bytesRead = br.Read(buffer, 0, buffer.Length)) > 0)
}
Catch {
Write-Host "Error occurred"
}
Finally {
if ($Fs -ne $null)
{
$Fs.Dispose()
}
}
}
return $null
}
$Context = New-Object Microsoft.SharePoint.Client.ClientContext($SiteURL)
$securePassword=ConvertTo-SecureString $SPOnlinePassword -AsPlainText -Force
$Context.Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($SPOnlineUserName, $securePassword)
foreach($fileName in Get-ChildItem $fileDirectory)
{
$UpFile = UploadFileInSlice -ctx $Context -libraryName $libraryName -fileName $fileName.FullName
$Context.Dispose();
}

Once you run the PowerShell script, you can see the files uploaded to SharePoint online document library.

Method invocation failed because [Microsoft.SharePoint.Client.File] does not contain a method named StartUpload

While uploading large files to SharePoint online document library using PowerShell in SharePoint online, I got the below error which says: Method invocation failed because [Microsoft.SharePoint.Client.File] does not contain a method named ‘StartUpload’

Method invocation failed because [Microsoft.SharePoint.Client.File] does not contain a method named StartUpload

Below is the code we can write using Visual Studio Code or Windows PowerShell ISE.

$ContentStream = New-Object System.IO.MemoryStream

# Add an empty file.
$fileInfo = New-Object Microsoft.SharePoint.Client.FileCreationInformation
$fileInfo.ContentStream = $ContentStream
$fileInfo.Url = $UniqueFileName
$fileInfo.Overwrite = $true
$Upload = $Docs.RootFolder.Files.Add($fileInfo)
$ctx.Load($Upload)

# Start upload by uploading the first slice.

$s = [System.IO.MemoryStream]::new($buffer)

# Call the start upload method on the first slice.
$BytesUploaded = $Upload.StartUpload($UploadId, $s)
$ctx.ExecuteQuery()

The error was coming in the below line:

$BytesUploaded = $Upload.StartUpload($UploadId, $s)

Here I was using the 15 versions of Microsoft.SharePoint.Client.dll and Microsoft.SharePoint.Client.Runtime.dll. Here you need to use the 16 versions of both the dlls. Once I replaced with updated 16 versions of client dlls, it worked fine.

Read some PowerShell tutorials:

• Increase SharePoint Online Storage Quota using PowerShell
• Create web application in SharePoint 2016 from Central Administration and using PowerShell
• Delete list items created before N days using PowerShell in SharePoint Online
• Bulk SharePoint Online Site Collection Creation using PowerShell
• Show hidden features in SharePoint 2013 using PowerShell
• SharePoint Online CAML Query Filter Example with CSOM

I hope this PowerShell script will be helpful to upload large files to SharePoint online document library.