Skip to content

Latest commit

 

History

History
102 lines (76 loc) · 5.3 KB

bulk_download_using_wget.md

File metadata and controls

102 lines (76 loc) · 5.3 KB
Raw

How to bulk download LP DAAC data using wget

This guide shows how to bulk download LP DAAC data using wget from the command line. The wget command is a tool developed by the GNU Project to download files from the web. Wget allows you to automate retrieving content and files from web servers using a command-line interface.

Requirements:

Method 1: Authenticate using .wgetrc file

Step 1: Save the Download Link(s)

Save download links for your data as a text file using Nasa Earthdata Search or Common Metadata Repository (CMR) API. Follow the steps in the Earthdata Search guide to find your data and save the download links. If you prefer to use an API to find your data and save the download links, a tutorial on how to use the CMR API can be found here.

Step 2: Set up a .wgetrc file for Authentication

Set up a .wgetrc file in your home directory.

  • Manual setup:

    • Download the .wgetrc template file and save it in your home directory.
      • Open the .wgetrc file in a text editor and replace <USERNAME> with your NASA Earthdata Login username and <PASSWORD> with your NASA Earthdata Login password.

  • Create .wgetrc file from Command Line:

    • To Create a .wgetrc file, enter the following in Terminal.

      • Windows

        NUL >> .wgetrc

      • MacOS or Linux

        To Create a .netrc file, enter the following in the command line.

        touch .wgetrc | chmod og-rw .wgetrc

    • To insert your NASA Earthdata Login username and password into the file, enter the following in the Command Prompt and replace your username and password.

      echo http-user=Insert_Your_Username >> .wgetrc | echo http-password=Insert_Your_Password >> .wgetrc

Step 3: Download LP DAAC Data

You should now be able to run wget commands to download data directly from the LP DAAC.

  • Navigate to the directory you want to save the data using cd Insert_Your_Directory.
  • To download a single file, replace the Insert_the_Download_Link in the command below with the URL to the data file you wish to download 
    wget Insert_the_Download_Link
    Example: 
    wget https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T12SWF.2023189T175919.v2.0/HLS.S30.T12SWF.2023189T175919.v2.0.B08.tif
  • To download multiple files, replace Insert_Text_File in the command below with the full path to the text file saved previously in Step 1. 
    wget -i Insert_Text_File
    Example: 
    wget -i data/Granule-DownloadLinks.txt

Method 2: Authenticate Manually

If you prefer to not create .wgetrc file for authentication, you can make wget command to work by passing authentications manually.

Download LP DAAC Data

  • Navigate to the directory you want to save the data using cd Insert_Your_Directory.
  • Replace your Earthdata login username with "Insert_Your_Username" below.

    • To download a single file, replace the Insert_the_Download_Link in the command below with the URL to the data file you wish to download

      wget --http-user=Insert_Your_Username --ask-password --keep-session-cookies Insert_the_Download_Link

      Example:

      wget --http-user=MYUSERNAME --ask-password --keep-session-cookies https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T12SWF.2023189T175919.v2.0/HLS.S30.T12SWF.2023189T175919.v2.0.B08.tif

    • To download multiple files, replace Insert_Your_Username with your Earthdata Login username in the command below. Also, replace Insert_Text_File with the full path to the text file saved previously in Step 1. You will be asked to enter your password (i.e., you Earthdata Login password) after running the command. You'll press enter again to download your files.

      wget --http-user=Insert_Your_Username --ask-password --keep-session-cookies -i Insert_Text_File

      Example:

      wget --http-user=MYUSERNAME --ask-password --keep-session-cookies -i data/Granule-DownloadLinks.txt
    Alternatively, you can replace --ask-password with --http-passwd=Insert_Your_Password and enter your password directly in the command line.

Contact Info:

Email: [email protected]
Voice: +1-866-573-3222
Organization: Land Processes Distributed Active Archive Center (LP DAAC)¹
Website: https://lpdaac.usgs.gov/
Date last modified: 07-12-2023

¹Work performed under USGS contract G15PD00467 for NASA contract NNG14HH33I.